This is a calibration reading list maintained by the Beijing Language and Culture University Intelligent Computer-Assisted Language Learning (ICALL) Research Group.

• History
• Machine Learning
• Recalibration
• Ensemble
• Sampling
• Others
• Bias
• Cutting Edge

• COOKE, W.E. 1906. Forecasts and Verifications in Western Australia. Monthly Weather Review, 34, 23-24.

• COOKE, W.E. 1906. Weighting Forecasts. Monthly Weather Review, 34, 274-275.

• G. W. Brier. 1944. Verification of a Forecaster's Confidence and the Use of Probability Statements in Weather Forecastin. Research Paper No. 16, U.S. Weather Bureau, Washington, D.C.

• G. W. Brier. 1950. Verification of forecasts expressed in terms of probability. Monthly Weather Review.

• Ayer, M., Brunk, H., Ewing, G., Reid, W., & Silverman, E. 1955. An empirical distribution function for sampling with incomplete information. Annals of Mathematical Statistics, 5, 641--647.

• Allan H. Murphy. 1973. A New Vector Partition of the Probability Score. Journal of Applied Meteorology (1962-1982) Vol. 12, No. 4 (June 1973), pp. 595-600 (6 pages).

• Allan H. Murphy and Robert L. Winkler. 1977. Reliability of Subjective Probability Forecasts of Precipitation and Temperature. Journal of the Royal Statistical Society. Series C (Applied Statistics) Vol. 26, No. 1 (1977), pp. 41-47 (7 pages).

• Allan H. Murphy and Robert L. Winkler. 1984. Probability Forecasting in Meterology. Journal of the American Statistical Association , Sep., 1984, Vol. 79, No. 387(Sep., 1984), pp. 489-500.

• Alexandru Niculescu-Mizil , Rich Caruana. 2005. Predicting Good Probabilities With Supervised Learning. In Proceedings of the 22nd international conference on Machine learning, August, 2005, Pages 625–632.

• Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 1321–1330.

• Rafael Müller, Simon Kornblith, Geoffrey Hinton. 2019. When Does Label Smoothing Help?. In NeurIPS 2019.

• Aviral Kumar, Sunita Sarawagi. 2019. Calibration of Encoder Decoder Models for Neural Machine Translation. In ICLR Debugging Machine Learning Models Workshop.

• Shuo Wang, Zhaopeng Tu, Shuming Shi, Yang Liu. 2020. On the Inference Calibration of Neural Machine Translation. In Proceedings of ACL 2020.

• Shrey Desai, Greg Durrett. 2020. Calibration of Pre-trained Transformers. In Proceedings of EMNLP 2020.

• John C. Platt. 2000. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Advances in Large Margin Classi-fiers, pages 61–74.

• Zadrozny, Bianca and Elkan, Charles. 2001. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In ICML, pp. 609–616, 2001.

• Zadrozny, Bianca and Elkan, Charles. Transforming classifier scores into accurate multiclass probability estimates. In KDD, pp. 694–699, 2002.

• Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining Well Calibrated Probabilities Using Bayesian Binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence.

• Balaji Lakshminarayanan, Alexander Pritzel, Charles Blundell. 2017. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In NIPS 2017.

• Steven Reich, David Mueller, Nicholas Andrews. 2020. Ensemble Distillation for Structured Prediction: Calibrated, Accurate, Fast---Choose Three. In EMNLP 2020.

• Yarin Gal and Zoubin Ghahramani. 2016. Dropout as abayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059.

• Alex Kendall and Yarin Gal. 2017. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?. In NIPS 2017.

• Simon A. A. Kohl, Bernardino Romera-Paredes, Clemens Meyer, Jeffrey De Fauw, Joseph R. Ledsam, Klaus H. Maier-Hein, S. M. Ali Eslami, Danilo Jimenez Rezende, Olaf Ronneberger. 2018. A Probabilistic U-Net for Segmentation of Ambiguous Images. In NeurIPS 2018.

• Simon A. A. Kohl, Bernardino Romera-Paredes, Klaus H. Maier-Hein, Danilo Jimenez Rezende, S. M. Ali Eslami, Pushmeet Kohli, Andrew Zisserman, Olaf Ronneberger. 2019. A Hierarchical Probabilistic U-Net for Modeling Multi-Scale Ambiguities. In Arxiv.

• Christian F. Baumgartner, Kerem C. Tezcan, Krishna Chaitanya, Andreas M. Hötker, Urs J. Muehlematter, Khoschy Schawkat, Anton S. Becker, Olivio Donati, Ender Konukoglu. 2019. PHiSeg: Capturing Uncertainty in Medical Image Segmentation. In MICCAI 2019.

• Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson, Gianluca Bontempi. 2015. Calibrating Probability with Undersampling for Unbalanced Classification. 2015 IEEE Symposium Series on Computational Intelligence (SSCI), Cape Town, South Africa.

• Volodymyr Kuleshov and Percy S Liang. 2015. Calibrated structured prediction. In Advances in Neural Information Processing Systems, pages 3474–3482.

• Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, Peter Flach. 2019. Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration. In NeurIPS 2019.

• Khanh Nguyen and Hong Yu. 2020. Calibrating Structured Output Predictors for Natural Language Processing. In Proceedings of ACL 2020.

• Carsten F. Dormann. 2020. Calibration of probability predictions from machine‐learning and statistical models. In Global Ecology and Biogeography.

• Amir Rahimi, Amirreza Shaban, Ching-An Cheng, Richard Hartley, Byron Boots. 2020. Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks. In NeurIPS 2020.

• Taehee Jung, Dongyeop Kang, Hua Cheng, Lucas Mentch, Thomas Schaaf. 2020. Posterior Calibrated Training on Sentence Classification Tasks. In ACL 2020.

• Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, Richard Hartley. 2020. Calibration of Neural Networks using Splines. In Arxiv.

• Tim Leathart, Maksymilian Polaczuk. 2020. Temporal Probability Calibration. In Arxiv.