Efficient leave-one-out cross-validation and WAIC for fitted Bayesian models

• CRAN install.packages("loo")

• GitHub devtools::install_github("jgabry/loo")

Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps.

This package implements the fast and stable computations for LOO and WAIC from Efficient leave-one-out cross-validation and WAIC for evaluating fitted Bayesian models. From existing posterior simulation draws, we compute LOO using Pareto smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparing predictive errors between two models.

Aki Vehtari, Andrew Gelman, Jonah Gabry

Corresponding MATLAB code can be found at https://github.com/avehtari/MatlabPSIS