Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> November 2018 <s> Abstract We consider the problem of accounting for model uncertainty in linear regression models. Conditioning on a single selected model ignores model uncertainty, and thus leads to the underestimation of uncertainty when making inferences about quantities of interest. A Bayesian solution to this problem involves averaging over all possible models (i.e., combinations of predictors) when making inferences about quantities of interest. This approach is often not practical. In this article we offer two alternative approaches. First, we describe an ad hoc procedure, “Occam's window,” which indicates a small set of models over which a model average can be computed. Second, we describe a Markov chain Monte Carlo approach that directly approximates the exact solution. In the presence of model uncertainty, both of these model averaging procedures provide better predictive performance than any single model that might reasonably have been selected. In the extreme case where there are many candidate predictors but ... <s> BIB001 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> November 2018 <s> Standard statistical practice ignores model uncertainty. Data analysts typically select a model from some class of models and then proceed as if the selected model had generated the data. This approach ignores the uncertainty in model selection, leading to over-confident in- ferences and decisions that are more risky than one thinks they are. Bayesian model averaging (BMA) provides a coherent mechanism for ac- counting for this model uncertainty. Several methods for implementing BMA have recently emerged. We discuss these methods and present a number of examples. In these examples, BMA provides improved out-of- sample predictive performance. We also provide a catalogue of currently available BMA software. <s> BIB002 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> November 2018 <s> Ensembles used for probabilistic weather forecasting often exhibit a spread-error correlation, but they tend to be underdispersive. This paper proposes a statistical method for postprocessing ensembles based on Bayesian model averaging (BMA), which is a standard method for combining predictive distributions from different sources. The BMA predictive probability density function (PDF) of any quantity of interest is a weighted average of PDFs centered on the individual bias-corrected forecasts, where the weights are equal to posterior probabilities of the models generating the forecasts and reflect the models’ relative contributions to predictive skill over the training period. The BMA weights can be used to assess the usefulness of ensemble members, and this can be used as a basis for selecting ensemble members; this can be useful given the cost of running large ensembles. The BMA PDF can be represented as an unweighted ensemble of any desired size, by simulating from the BMA predictive distribution. The BMA predictive variance can be decomposed into two components, one corresponding to the between-forecast variability, and the second to the within-forecast variability. Predictive PDFs or intervals based solely on the ensemble spread incorporate the first component but not the second. Thus BMA provides a theoretical explanation of the tendency of ensembles to exhibit a spread-error correlation but yet be underdispersive. The method was applied to 48-h forecasts of surface temperature in the Pacific Northwest in January– June 2000 using the University of Washington fifth-generation Pennsylvania State University–NCAR Mesoscale Model (MM5) ensemble. The predictive PDFs were much better calibrated than the raw ensemble, and the BMA forecasts were sharp in that 90% BMA prediction intervals were 66% shorter on average than those produced by sample climatology. As a by-product, BMA yields a deterministic point forecast, and this had root-mean-square errors 7% lower than the best of the ensemble members and 8% lower than the ensemble mean. Similar results were obtained for forecasts of sea level pressure. Simulation experiments show that BMA performs reasonably well when the underlying ensemble is calibrated, or even overdispersed. <s> BIB003 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> November 2018 <s> Abstract Ensemble prediction systems typically show positive spread-error correlation, but they are subject to forecast bias and dispersion errors, and are therefore uncalibrated. This work proposes the use of ensemble model output statistics (EMOS), an easy-to-implement postprocessing technique that addresses both forecast bias and underdispersion and takes into account the spread-skill relationship. The technique is based on multiple linear regression and is akin to the superensemble approach that has traditionally been used for deterministic-style forecasts. The EMOS technique yields probabilistic forecasts that take the form of Gaussian predictive probability density functions (PDFs) for continuous weather variables and can be applied to gridded model output. The EMOS predictive mean is a bias-corrected weighted average of the ensemble member forecasts, with coefficients that can be interpreted in terms of the relative contributions of the member models to the ensemble, and provides a highly competiti... <s> BIB004 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> November 2018 <s> In this paper, the Genetic Algorithms (GA) and Bayesian Model Averaging (BMA) were used to simultaneously conduct calibration and uncertainty analysis for the Soil and Water Assessment Tool (SWAT). In this combined method, several SWAT models with different structures are first selected; next GA is used to calibrate each model using observed streamflow data; finally, BMA is applied to combine the ensemble predictions and provide uncertainty interval estimation. This method was tested in two contrasting basins, the Little River Experimental Basin in Georgia, USA, and the Yellow River Headwater Basin in China. The results obtained in the two case studies show that this combined method can provide deterministic predictions better than or comparable to the best calibrated model using GA. The 66.7% and 90% uncertainty intervals estimated by this method were analyzed. The differences between the percentage of coverage of observations and the corresponding expected coverage percentage are within 10% for both calibration and validation periods in these two test basins. This combined methodology provides a practical and flexible tool to attain reliable deterministic simulation and uncertainty analysis of SWAT. <s> BIB005 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> November 2018 <s> We extend ensemble Bayesian model averaging (EBMA) for application to binary outcomes and illustrate EBMA’s ability to aid scholars in the social sciences to make more accurate forecasts of future events. In essence, EBMA improves prediction by pooling information from multiple forecast models to generate ensemble predictions similar to a weighted average of component forecasts. The weight assigned to each forecast is calibrated via its performance in some training period. The aim is not to choose some “best” model, but rather to incorporate the insights and knowledge implicit in various forecasting efforts via statistical postprocessing. After presenting the method, we show that EBMA increases the accuracy of out-of-sample forecasts relative to component models in three applied examples: predicting the occurrence of insurgencies around the Pacific Rim, forecasting vote shares in U.S. presidential elections, and predicting the votes of U.S. Supreme Court justices. <s> BIB006 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> November 2018 <s> We review the Akaike, deviance, and Watanabe-Akaike information criteria from a Bayesian perspective, where the goal is to estimate expected out-of-sample-prediction error using a bias-corrected adjustment of within-sample error. We focus on the choices involved in setting up these measures, and we compare them in three simple examples, one theoretical and two applied. The contribution of this paper is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice. <s> BIB007 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> November 2018 <s> This paper offers a theoretical explanation for the stylized fact that forecast combinations with estimated optimal weights often perform poorly in applications. The properties of the forecast combination are typically derived under the assumption that the weights are fixed, while in practice they need to be estimated. If the fact that the weights are random rather than fixed is taken into account during the optimality derivation, then the forecast combination will be biased (even when the original forecasts are unbiased), and its variance will be larger than in the fixed-weight case. In particular, there is no guarantee that the ‘optimal’ forecast combination will be better than the equal-weight case, or even improve on the original forecasts. We provide the underlying theory, some special cases, and a numerical illustration. <s> BIB008
MODEL AVERAGING IN ECOLOGY 489 model weights). Only these approaches are at least trying to estimate optimal weights for minimizing predictive error. The interactive tool we provide ( Fig. 3 ) allows readers to explore this issue in a simple two-model case. It shows that, in this simple case, estimating weights substantially reduces the parameter space where model averaging is superior to the best single model. Thus, the bias-variance trade-off applies also to model averaging, in the sense that weight estimation introduces additional parameters and therefore higher model complexity to the analysis. It is therefore important to think carefully about when to use model averaging, as it can add unnecessary complexity. Uncertainty about the optimal weights does not imply that estimated weights are of no use, or that the use of arbitrary weights (e.g., equal weights) is generally superior. While uncertainty in estimated weights increases prediction error, the ability to statistically downweight or wholly remove unsuitable models from the prediction set is a substantial benefit. In BIB008 and similar simulations, all models considered are "alright" (bias-free and with similar prediction variance), which obviously need not be the case in practical applications. Thus, the question is not if estimated model weights are useful in general, but how useful they are beyond their function of filtering out inferior models from the average. We believe there is a benefit beyond this filter function, but we recognize that there is a need for further research to better demonstrate this benefit, and understand when it occurs. sufficiently peaked by the calibration step, this approach should eliminate any complication resulting from the prior choice (for an ecological example see van Oijen et al. 2013 ). Bayesian-flavoured approaches.-Apart from the natural Bayesian average (see also  , there are a number of other approaches that are connected to or inspired by Bayesian thinking. In a set of influential publications, BIB001 , BIB002 , and BIB003 introduced post hoc Bayesian model averaging, i.e., for vectors of predictions from already fitted models. The key idea is to iteratively estimate the proportion of times a model would yield the highest likelihood within the set of models (through expectation maximization, see Appendix S1.5.2 for details), and use this proportion as model weight. In the spirit of the inventors, we refer to this approach as Bayesian model averaging using Expectation-Maximization (BMA-EM), but place it closer to a frequentist than a Bayesian approach, as the models were not necessarily (and in none of their examples) fitted within the Bayesian framework. It has been used regularly, often for process models (e.g., BIB004 BIB005 ), where an rjMCMC procedure would require substantial programming work at little perceived benefit, but also in data-poor situations in the political sciences BIB006 ). Chickering and Heckerman (1997) investigate approximations of the marginal likelihood in Eq. 9, such as the Bayesian Information Criterion (BIC, as defined in the next section; see also Appendix S1.5.3) and find them to work well for model selection, but not for model averaging. In contrast, Kass and Raftery (1995:778) state the e BIC is an acceptable approximation of the Bayes factor, and hence suitable for model averaging, despite being biased even for large sample sizes. These approximations may be improved when using more complex versions of BIC (SPBIC and IBIC;  . The "widely applicable information criterion" WAIC (Watanabe 2010 and an equivalent WBIC Watanabe 2013) are motivated and actually analytically derived in a Bayesian framework BIB007 . With an uninformative prior, it can be seen as a variation of AIC (see next section). The WAIC is computed, for each model, from two terms BIB007 : (1) the log pointwise predicted density (lppd) across the posterior simulations for each of the n predicted values, defined as lppd ¼ log Q n i¼1 p posterior ðy i Þ; and (2) a bias-correction term p WAIC ¼ P n i¼1 varðlogðpðy i jh s ÞÞÞ, where var is the sample variance over all S samples of the posterior distributions of parameters h. The WAIC is then defined as WAIC ¼ À2lppd þ 2p WAIC . In other words, the WAIC is the likelihood of observing the data under the posterior parameter distributions, corrected by a penalty of model complexity proportional to the variance of these likelihoods across the MCMC samples. Model weights are computed from WAIC analogously to Eq. 11 below. MODEL AVERAGING IN ECOLOGY 497 model weights, as these "model selection" outcomes of model averaging fared worse than those that kept all models in the set (equal weight, leave-one-out, and cos-squared). Second, the best two model averaging algorithms in this case study, apart from the median where varying weights are used, identified an approximately equal weighting as TABLE 2. Model weights (averaged across 100 repetitions) given to the 16 linear regression models of case study 1 by different weighting methods (see Table 1 for abbreviations), arranged by increasing prediction error (last column, median across replications). 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15 16 RMSE rjMCMC median 0.00 0.01 0.00 0.11 0.00 0.00 0.08 0.11 0.00 0.14 0.00 0.09 0.14 0.13 0.10 0.09 1.069 BIC 0.00 0.01 0.00 0.18 0.00 0.03 0.17 0.04 0.00 0.19 0.00 0.04 0.24 0.05 0.05 0.01 1.074 0.00 0.01 0.00 0.11 0.00 0.00 0.08 0.11 0.00 0.14 0.00 0.09 0.14 0.13 0.10 0.09 1.076 Boot 0.00 0.01 0.00 0.15 0.00 0.04 0.17 0.03 0.00 0.16 0.00 0.08 0.22 0.04 0.07 0.03 1.076 AIC 0.00 0.00 0.00 0.13 0.00 0.02 0.13 0.08 0.00 0.14 0.00 0.08 0.18 0.09 0.09 0.05 1.077 WAIC 0.00 0.00 0.00 0.13 0.00 0.02 0.11 0.09 0.00 0.14 0.00 0.08 0.16 0.10 0.11 0.06 1.078 MMA 0.00 0.00 0.00 0.13 0.00 0.02 0.12 0.08 0.00 0.14 0.00 0.09 0.18 0.10 0.10 0.06  181 Minimal variance -1.15 0.42 0.19 0.00 0.64 0.00 0.00 0.00 0.91 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.208 Cos-squared 0.00 0.00 0.30 0.00 0.21 0.21 0.02 0.01 0.00 0.00 0.24 0.00 0.00 0.00 0.01 0.00 1.209 Notes: Only the best (m10) and the full model are shown from the 16 candidate models. LOO-CV, leave-one-out cross-validation using R 2 or RMSE as measure of model performance. For code see case study 1 in Data SI. †Weights not available, as different models contribute to the median at each replication. ‡Prediction from individual model. §Weights are variable. LM and rF refer to a linear model and a Random Forest as supra-model, respectively. optimal strategy. That is somewhat surprising, given that SVM performed relatively poorly (and was excluded by BMA-EM, but favored by cos-squared as a more independent contribution). The likely reason of high weights for the poor SVM is that averaging-in less correlated predictions reduces covariances in Eq. 5. The good performance of the median in both case studies suggests that using the central value of each prediction, rather than give constant weights to the model itself, may be even more effective in reducing variance and thus prediction error. Further research is needed to clarify if this principle is indeed valid across many applications.