Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

The pre-commit is failing because the notebook (examples/case_studies/spline.ipynb) doesn't match the corresponding jupytext file (myst_nbs/case_studies/spline.myst.md). I think you should be able to install the jupytext extension (https://jupytext.readthedocs.io/en/latest/install.html), restart jupyter, open the notebook, save it, and then commit/push.

Done, I re-run and re-commited after installing jupytext. thanks!

Maybe you have to make some minor change (e.g., re-run the last cell)? When you go to commit, you should see changes to myst_nbs/case_studies/spline.myst.md and those changes need to be committed/pushed.

Thanks! I re-ran the notebook before I committed, but I checked and spline.myst.md is unchanged. I did some searching and found this on Stackoverflow, so I re-install jupytext using conda install jupytext[myst] (install was successful). I made a small change, re-ran the notebook, saved - still no changes to spline.myst.md

Is there anything else I need to do to enable this auto-update of *.myst.md?

Strange. I guess you can check to see if the jupytext extension was installed (see here). You can also try running the pre-commit on this notebook locally (see here).

I do have Jupytext installed. I was able to save a new spline.myst.md file by editing notebook metadata as follows

"jupytext": {
    "formats": "ipynb,.myst.md:myst"
  }

This saves the splines.myst.md in the same directory as the splines.ipynb, so I had to manually move it to correct folder. Do you know how Jupytext destination folder is controlled in pymc-examples? I found Jupytext manual on this, but it looks like there are multiple options. Thanks!

@kuvychko you should not update the notebook metadata with this. The jupytext config is set at the project level with https://github.com/pymc-devs/pymc-examples/blob/main/.jupytext.toml. As jupytext is not the only thing that can fail pre-commit, I'd recommend running pre-commit directly.

Try pre-commit run --file examples/case_studies/spline.ipynb then add and commit all the updated files. Also, I know we still lack docs on this, especially regarding the jupytext syncronization, so it is also fine if you want us to take care of the pre-commit and myst.

@OriolAbril gotcha, I'll install pre-commit and validate locally and roll back metadata change. I'd like to figure this out since I want to keep contributing to example notebooks. Thanks for working through this with me! Question: do I need to install Sphynx for auto-compile of myst.md to work correctly?

No, you do not need sphinx. Sphinx is only needed to generate the documentation. The .myst.md files are an alternative representation with basically the same content, but encoded in a more reader-friendly manner.

The reason behind the myst+ipynb duplication is that especially for reviewing the PRs (but also when working depending on what you are doing) it is necessary to have it. With all the improvements and changes I explained in the webinar, we can now use many "hidden" features of notebooks to customize and format the rendered pages. I didn't go into these features during the webinar though.

One example is using "cell tags" to hide some of the cells, this is useful if the cell only has plotter helpers to allow interested readers to click to expand the cell and read the code but to reduce the wall of text and code complexity for newcomers or people skimming the notebook; what is important to read and understand the notebook is seeing the plot and reading the explanation that goes along it, not the specific matplotlib code used to generate it. The moderation analysis notebook (among others) uses this:

• Rendered page: https://www.pymc.io/projects/examples/en/latest/case_studies/moderation_analysis.html
• ipynb source: https://github.com/pymc-devs/pymc-examples/blob/main/examples/case_studies/moderation_analysis.ipynb
• myst source: https://github.com/pymc-devs/pymc-examples/blob/main/myst_nbs/case_studies/moderation_analysis.myst.md

It is easy to see the :tags: [hide-input] pieces in the myst representation, but it is not possible with the jupyter one.

Another example is formatting generated plots as figures. @reshamas is starting on this in #402. There you can see that the ipynb raw diff is not available, whereas the myst diff clearly shows the metadata that defines the caption and alt text for the plot (you have to scroll down a bit but that is all).

At the same time however, we can't stop using ipynb files because the notebooks take too long to run, so we need the outputs to be stored in the file so we don't need to run the notebooks to generate the website.

I see! Makes sense, thanks for the explanation!

When I run pre-commit now I get this message (among many): [jupytext] Updating examples/case_studies/spline.md. pre-commit fails for jupytext since spline.myst.md is not being updated. It looks like during pre-commit a spline.md file is generated in examples/case_studies. Any thoughts on how to fix it? Other checks are passing.

I think the file level metadata takes priority over the project level config. So instead of generating the copy in the myst files like the others, as the ipynb has specific sync metadata it follows the instructions there instead

make sure neither the ipynb nor myst files have the sync metadata you previously added before running pre-commit

Also, pre-commit will fail al first, but it will generate or update the files needed for the check to pass. You can then git add them and push, and ci should pass

Also, I know we still lack docs on this, especially regarding the jupytext syncronization, so it is also fine if you want us to take care of the pre-commit and myst.

Indeed. If it helps @kuvychko , this same synchronization issue was a major headache for the core developers when they were trying to update notebooks before the v4.0 release. So if they had trouble, it's definitely not something that you are doing.

@cluhmann - good to know! :-) I must have messed up notebook metadata. I wound up rolling back my changes, and making them again starting with a pristine copy of splines.ipynb. @OriolAbril - thanks for the tip of running pre-commit twice! I got it to pass locally. Hope the change looks good. Thanks for working through this with me!

Success! Thank you for your contribution @kuvychko!