Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix notebooks #35

Merged
merged 18 commits into from
Aug 19, 2024
Merged

Fix notebooks #35

merged 18 commits into from
Aug 19, 2024

Conversation

ederag
Copy link
Contributor

@ederag ederag commented Aug 15, 2024

The julia-1.8 removal from CI comes from #34 (comment) (just forgot to push back then).

Until 70197f4 can be reverted (or improved upon),
one needs a valid project.
For notebooks development, here is how I proceed

# moving to the notebooks folder is important, for the notebook to be edited in-place rather than copied
cd samples/notebooks/
julia --project --threads=auto

then

julia> import OpticSim.NotebooksUtils as NB

julia> NB.run_sample("EmittersIntro.jl")

With the other changes the Pluto notebook
samples/notebooks/EmittersIntro.jl now executes without any error,
and all Makie-related warnings have been silenced as well.

samples/notebooks/BasicCSG.jl still errors with

MethodError: no method matching leaf(::OpticSim.CSGGenerator{Float64}, ::OpticSim.Geometry.Transform{Float64})

Closest candidates are:

leaf(!Matched::OpticSim.ParametricSurface{T}, ::OpticSim.Geometry.Transform{T}) where T<:Real

@ OpticSim ~/share/prog/julia/dev/OpticSim.jl/src/Geometry/CSG/CSG.jl:126

And samples/notebooks/Samples.jl has two errors:

UndefVarError: `GridRectOriginPoints` not defined

and

MethodError: no method matching OpticSim.Emitters.Sources.CompositeSource(...
You might have used a 2d row vector where a 1d column vector was required.
Note the difference between 1d column vector [1,2,3] and 2d row vector [1 2 3].
You can convert to a column vector with the vec() function.

Those errors seem related to changes in OptimSim rather than Makie or julia, so I'd leave them for later.

ederag added 13 commits August 15, 2024 19:16
The Pluto built-in package manager could not arrive at a solution
because the Opticsim release it picks up (0.5.3)
does not have the required fixes.
Bonito now sets the port automatically.
The replacement of Vis.scene is then no longer necessary,
as Vis.scene bypasses the buttons layout stuff when
current mode is :pluto or :docs.
Minimal change to adapt to Makie 0.20.
Minimal change to adapt to Makie 0.20,
while keeping the current OpticSim.Vis API.
This line errored with Makie 0.20,
and the comment said it didn't seem to work anyway.
The Pluto built-in package manager could not arrive at a solution
because the Opticsim release it picks up (0.5.3)
does not have the required fixes.
The replacement of Vis.scene is then no longer necessary,
as Vis.scene bypasses the buttons layout stuff when
current mode is :pluto or :docs.
The Pluto built-in package manager could not arrive at a solution
because the Opticsim release it picks up (0.5.3)
does not have the required fixes.
The replacement of Vis.scene is then no longer necessary,
as Vis.scene bypasses the buttons layout stuff when
current mode is :pluto or :docs.
ubuntu-18.04 is no longer available on github CI.
@ederag
Copy link
Contributor Author

ederag commented Aug 18, 2024

Documentation failed because there were no runners:

The ubuntu-18.04 environment is deprecated, consider switching to ubuntu-20.04(ubuntu-latest), or ubuntu-22.04 instead.

=> updated to ubuntu-latest, and also updated julia to 1.10 for the docs as well.

@ederag
Copy link
Contributor Author

ederag commented Aug 18, 2024

Note: those are just notes explaining why I'm making seemingly strange changes,
no need to waste time reading this comment further.

Doc CI fails with

ERROR: Unsatisfiable requirements detected for package SHA [ea8e919c]:
 SHA [ea8e919c] log:
 ├─possible versions are: 1.1.1 or uninstalled
 └─restricted to versions 0.7.0-0.7 by OpticSim [24114763] — no versions left

From JuliaCrypto/SHA.jl#87 (comment):

The latest version of this package is 0.7. You cannot install SHA@0.7 on Julia 1.6 and 1.7 as these versions already include SHA.jl as a stdlib

[see below, on 1.10 it is possible to install SHA@0.7, and it is still the latest version]

So let's remove SHA from compat ?
Just in case there would have been a UUID change, remove and add again:

(OpticSim) pkg> rm SHA
    Updating `~/share/prog/julia/dev/OpticSim.jl/Project.toml`
  [ea8e919c] - SHA v0.7.0
  No Changes to `~/share/prog/julia/dev/OpticSim.jl/Manifest.toml`

(OpticSim) pkg> add SHA
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `~/share/prog/julia/dev/OpticSim.jl/Project.toml`
  [ea8e919c] + SHA v0.7.0
  No Changes to `~/share/prog/julia/dev/OpticSim.jl/Manifest.toml`

Nope, 0.7.0 is still the most recent SHA version.

Besides, it is possible to use SHA@0.7 (it is actually the one bundled with julia-1.10):

(@v1.10) pkg> activate --temp
  Activating new project at `/tmp/jl_Aex54p`

julia> using SHA

julia> pkgversion(SHA)
v"0.7.0"

(jl_Aex54p) pkg> add SHA@0.7

Works fine.

Then why did the CI say "possible versions are: 1.1.1 or uninstalled" ?
Maybe a registry corruption ?

Let me push the change anyway to see if there is any difference.

EDIT: note that github shows the commits before this comment
(the display ordering is based on the commit authoring time, not the push time...)

Note:
  tests passed with julia-1.8 on linux,
  but 1.8 was removed from CI recently.
  brianguenter#34 (comment)
@ederag
Copy link
Contributor Author

ederag commented Aug 18, 2024

Still fails;
it looks like the CI process runs julia 1.1 instead of 1.10, despite a8be81e.

CI log
Run xvfb-run julia --project=docs/ -e '
   Cloning default registries into `~/.julia`
   Cloning registry from "https://github.com/JuliaRegistries/General.git"
25l    Fetching: [>                                        ]  0.0 %
    Fetching: [=====>                                   ]  10.1 %
    Fetching: [==============>                          ]  34.3 %
    Fetching: [===========================>             ]  66.9 %
    Fetching: [=====================================>   ]  90.8 %
25h     Added registry `General` to `~/.julia/registries/General`
 Resolving package versions...
┌ Warning: julia version requirement for package OpticSim not satisfied
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:154
ERROR: Unsatisfiable requirements detected for package SHA [ea8e919c]:
 SHA [ea8e919c] log:
 ├─possible versions are: 1.1.1 or uninstalled
 ├─restricted to versions * by OpticSim [24114763], leaving only versions 1.1.1
 │ └─OpticSim [24114763] log:
 │   ├─possible versions are: 0.6.0 or uninstalled
 │   └─OpticSim [24114763] is fixed to version 0.6.0
 └─found to have no compatible versions left with Bonito [824d6782] 
   └─Bonito [824d6782] log:
     ├─possible versions are: [3.0.0-3.0.7, 3.1.0-3.1.2] or uninstalled
     └─restricted to versions * by OpticSim [24114763], leaving only versions [3.0.0-3.0.7, 3.1.0-3.1.2]
       └─OpticSim [24114763] log: see above
Stacktrace:
 [1] #propagate_constraints!#61(::Bool, ::Function, ::Pkg.GraphType.Graph, ::Set{Int64}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/GraphType.jl:1007
 [2] propagate_constraints! at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/GraphType.jl:948 [inlined]
 [3] #simplify_graph!#121(::Bool, ::Function, ::Pkg.GraphType.Graph, ::Set{Int64}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/GraphType.jl:1462
 [4] simplify_graph! at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/GraphType.jl:1462 [inlined] (repeats 2 times)
 [5] resolve_versions!(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:371
 [6] resolve_versions! at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:315 [inlined]
 [7] #add_or_develop#63(::Array{Base.UUID,1}, ::Symbol, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1172
 [8] #add_or_develop at ./none:0 [inlined]
 [9] #add_or_develop#17(::Symbol, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:59
 [10] #add_or_develop at ./none:0 [inlined]
 [11] #add_or_develop#16 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:36 [inlined]
 [12] #add_or_develop at ./none:0 [inlined]
 [13] #add_or_develop#12 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:33 [inlined]
 [14] #add_or_develop at ./none:0 [inlined]
 [15] #develop#23 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:65 [inlined]
 [16] develop(::Pkg.Types.PackageSpec) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:65
 [17] top-level scope at none:0

Note the

┌ Warning: julia version requirement for package OpticSim not satisfied
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:154

and some julia/stdlib/v1.1/Pkg/ in the stacktrace.

The Documenter.jl documentation
puts the version inside single quotes. Let's try to do the same ?

Indeed, that's julia-actions/setup-julia#204

@ederag
Copy link
Contributor Author

ederag commented Aug 18, 2024

OK, it now goes further but got stuck for more than one hour after installing GLMakie:
Screenshot_20240818_201218

Let me revert 9a8d8f4 (it was not the culprit) to trigger the doc build again.

@ederag
Copy link
Contributor Author

ederag commented Aug 18, 2024

And it did it again:

CI log
# ... the log top was cropped
   Installed CRlibm_jll ─────────────────────── v1.0.1+0
   Installed Juno ───────────────────────────── v0.8.4
   Installed StringManipulation ─────────────── v0.3.4
   Installed WoodburyMatrices ───────────────── v1.0.0
   Installed SortingAlgorithms ──────────────── v1.2.1
   Installed ExpressionExplorer ─────────────── v1.0.2
   Installed ImageMorphology ────────────────── v0.4.5
   Installed WidgetsBase ────────────────────── v0.1.4
   Installed Luxor ──────────────────────────── v3.8.0
   Installed Pluto ──────────────────────────── v0.19.46
   Installed DelaunayTriangulation ──────────── v1.1.3
   Installed Makie ──────────────────────────── v0.20.10
    Updating `~/work/OpticSim.jl/OpticSim.jl/docs/Project.toml`
  [da1fd8a2] + CodeTracking v1.3.6
  [a93c6f00] + DataFrames v1.6.1
⌅ [e30172f5] + Documenter v0.26.3
⌅ [e9467ef8] + GLMakie v0.9.11
  [916415d5] + Images v0.26.1
⌅ [ee78f7c6] + Makie v0.20.10
  [24114763] + OpticSim v0.6.0 `~/work/OpticSim.jl/OpticSim.jl`
  [91a5bcdd] + Plots v1.40.5
  [90137ffa] + StaticArrays v1.9.7
  [1986cc42] + Unitful v1.21.0
    Updating `~/work/OpticSim.jl/OpticSim.jl/docs/Manifest.toml`
  [621f4979] + AbstractFFTs v1.5.0
  [1520ce14] + AbstractTrees v0.4.5
  [7d9f7c33] + Accessors v0.1.37
  [79e6a3ab] + Adapt v4.0.4
  [35492f91] + AdaptivePredicates v1.1.1
  [66dad0bd] + AliasTables v1.1.3
  [27a7e980] + Animations v0.4.1
  [ec485272] + ArnoldiMethod v0.4.0
  [4fba245c] + ArrayInterface v7.15.0
  [67c07d97] + Automa v1.0.4
  [13072b0f] + AxisAlgorithms v1.1.0
  [39de3d68] + AxisArrays v0.4.7
  [6e4b80f9] + BenchmarkTools v1.5.0
  [d1d4a3ce] + BitFlags v0.1.9
  [62783981] + BitTwiddlingConvenienceFunctions v0.1.6
  [824d6782] + Bonito v3.1.2
  [fa961155] + CEnum v0.5.0
  [2a0fbf3d] + CPUSummary v0.2.6
  [96374032] + CRlibm v1.0.1
  [336ed68f] + CSV v0.10.14
  [159f3aea] + Cairo v1.1.0
  [aafaddc9] + CatIndices v0.2.2
  [082447d4] + ChainRules v1.69.0
  [d360d2e6] + ChainRulesCore v1.24.0
  [fb6a15b2] + CloseOpenIntervals v0.1.13
  [aaaa29a8] + Clustering v0.15.7
  ✓ FiniteDiff
  ✓ StaticArrayInterface
  ✓ Zygote → ZygoteColorsExt
  ✓ FiniteDiff → FiniteDiffStaticArraysExt
  ✓ StaticArrayInterface → StaticArrayInterfaceStaticArraysExt
  ✓ NLSolversBase
  ✓ StaticArrayInterface → StaticArrayInterfaceOffsetArraysExt
  ✓ GtkObservables
  ✓ LineSearches
  ✓ TiledIteration
  ✓ CloseOpenIntervals
  ✓ LayoutPointers
  ✓ Optim
  ✓ MathOptInterface
  ✓ ImageView
  ✓ VectorizationBase
  ✓ GLPK
  ✓ NLopt → NLoptMathOptInterfaceExt
  ✓ JuMP
  ✓ Optim → OptimMOIExt
  ✓ SLEEFPirates
  ✓ Ipopt
  ✓ LazySets
  ✓ Plots
  ✓ Plots → FileIOExt
  ✓ Plots → UnitfulExt
  ✓ Plots → GeometryBasicsExt
  ✓ LoopVectorization
  ✓ LoopVectorization → SpecialFunctionsExt
  ✓ LoopVectorization → ForwardDiffExt
  ✓ ImageMorphology
  ✓ Makie
  ✓ ImageFiltering
  ✓ ImageDistances
  ✓ ImageSegmentation
  ✓ ImageCorners
  ✓ ImageQualityIndexes
  ✓ Images
WGLMakie Waiting for background task / IO / timer.
[pid 7269] waiting for IO to finish:
 Handle type        uv_handle_t->data
 timer              0x29c6c70->0x7f845c4ad480
This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
GLMakie Waiting for background task / IO / timer.
[pid 7268] waiting for IO to finish:
 Handle type        uv_handle_t->data
 timer              0x5ca5e90->0x7fd555699510
This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
  ✓ WGLMakie
  ✓ GLMakie

OK, this might be related to MakieOrg/Makie.jl#3766 (comment) which

shouldnt be happening anymore

So time to update to Makie 0.20 to the latest, 0.21.
Done locally, tests pass, but the notebooks error

Invalid attribute debug for plot type

I'll look into this in the next few days.

@brianguenter
Copy link
Owner

Do you want me to approve and merge this batch of changes in the meantime before you get the fix for Makie, or would you prefer to do it all at once later?

@ederag
Copy link
Contributor Author

ederag commented Aug 19, 2024

Do you want me to approve and merge this batch of changes in the meantime before you get the fix for Makie

In the current state (Makie 0.20) samples/notebooks/EmittersIntro.jl is fine,
only the doc CI hangs.
So if you are OK, it would be the right time to merge indeed,
before the upgrade to Makie 0.21.

EDIT (for other readers):
The no method matching leaf and GridRectOriginPoints errors described in #35 (comment)) are still there.

@brianguenter brianguenter merged commit 3f7d2a6 into brianguenter:main Aug 19, 2024
2 of 4 checks passed
@ederag ederag deleted the update_notebooks branch August 20, 2024 04:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants