Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update Model Averaging example to run on pymc v5 #67 #600

Merged
merged 3 commits into from
Aug 5, 2024

Conversation

alporter08
Copy link
Contributor

@alporter08 alporter08 commented Nov 17, 2023

update Model Averaging example to run on pymc v5

#67

Helpful links


📚 Documentation preview 📚: https://pymc-examples--600.org.readthedocs.build/en/600/

Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@alporter08 alporter08 marked this pull request as draft November 17, 2023 17:28
@alporter08 alporter08 changed the title update code to run on pymc 5 https://github.com/pymc-devs/pymc-examples/issues/67 update code to run on pymc 5 #67 Nov 17, 2023
@alporter08 alporter08 changed the title update code to run on pymc 5 #67 update Model Averaging example to run on pymc v5 #67 Nov 17, 2023
@alporter08
Copy link
Contributor Author

Hi @OriolAbril @cluhmann , I will look over the text to make sure it syncs with the code changes and look at the Jupyter Style guide. I also noticed there is an issue currently with the colors and axes in the final plot so will investigate that.

@cluhmann
Copy link
Member

Great, thanks again for your the help!

@alporter08 alporter08 force-pushed the model-averaging-nb-update branch from 748e8ad to 9319e14 Compare November 20, 2023 05:57
@alporter08 alporter08 marked this pull request as ready for review November 20, 2023 06:00
@alporter08
Copy link
Contributor Author

@OriolAbril @cluhmann I got the code to work with the new PyMC version, and the results match the old notebook. I also edited some of the text for clarity.

One point to mention however is that the notebook is supposed to demonstrate that the weighted model approach has more uncertainty than an individual model. But looking at the hdi, it is actually pretty much equal for the weighted approach and the compared individual model. I think this is because the model we are comparing the ensemble method to has a weight of one, and the other two models are effectively zero weights. So the ensemble method is really just one model, which we are comparing to itself. So maybe that should be specified in the commentary? Right now it says that the weighted model uncertainty is larger than the single model uncertainty, which isn't really the case.

@OriolAbril
Copy link
Member

Given this is taken from an example in the book, we should reproduce it as it is. Then if we deem necessary, add extra comments. I have just checked, and the book uses the pseudo-BMA method to compute the weights. We should then use the method argument when calling az.compare, and use those weights to generate the weighted posterior predictive samples.

Afterwards, it is probably a good idea to use az.compare with the stacking method and use the results+reference to the section on stacking to explain why it is ArviZ's default now. What do you think? Tagging also @aloctavodia who authored the notebook

@aloctavodia
Copy link
Member

I think is ok to merge, the mechanics is ok, even when the example itself is not that good. I can later work on a new version that better highlights the difference between doing model averaging and picking a single model.

@twiecki twiecki merged commit 3654d50 into pymc-devs:main Aug 5, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants