Fabian Krüger: Probabilistic Forecasting and Predictive Model Assessment based on MCMC Output
Fabian Krüger (Heidelberg Institute for Theoretical Studies) will give a 30 min seminar in the lunch area, 8th floor N.H. Abel's House at 14:45 September 29th.
Title: Probabilistic Forecasting and Predictive Model Assessment based on MCMC Output
Abstract: A rapidly growing literature uses Bayesian methods to produce probabilistic forecasts of meteorological, economic or financial variables. Thereby, the posterior predictive distribution of interest comes as a simulated sample, typically generated by a Markov chain Monte Carlo (MCMC) algorithm. We conduct a systematic analysis of how to make and evaluate probabilistic forecast distributions based on such MCMC output.
Utilizing the mathematical framework provided by the theory of proper scoring rules (e.g. Gneiting and Raftery, “Strictly Proper Scoring Rules, Prediction, and Estimation”, JASA, 2007), we develop a notion of consistency that allows for assessing the adequacy of methods for estimating the stationary distribution underlying the MCMC output. We then review asymptotic results that account for the salient features of Bayesian posterior simulators, and derive conditions under which choices from the literature satisfy this notion of consistency. Importantly, these conditions depend on the scoring rule being used, such that the choices of approximation method and scoring rule are heavily intertwined. While the popular logarithmic scoring rule often requires fairly stringent conditions, the continuous ranked probability score (Matheson and Winkler, “Scoring Rules for Continuous Probability Distributions”, Management Science, 1976) can be used under mild assumptions. These results are illustrated in a simulation study and an economic case study. A “mixture of parameters” estimator that efficiently exploits the parametric structure of the model performs well in these examples. By contrast, fully nonparametric kernel density estimation techniques prove to be problematic, particularly when used in conjunction with the logarithmic scoring rule.