Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> THE MATHEMATICS BEHIND MODEL AVERAGING <s> Two separate sets of forecasts of airline passenger data have been combined to form a composite set of forecasts. The main conclusion is that the composite set of forecasts can yield lower mean-square error than either of the original forecasts. Past errors of each of the original forecasts are used to determine the weights to attach to these two original forecasts in forming the combined forecasts, and different methods of deriving these weights are examined. <s> BIB001 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> THE MATHEMATICS BEHIND MODEL AVERAGING <s> We argue that model selection uncertainty should be fully incorporated into statistical inference whenever estimation is sensitive to model choice and that choice is made with reference to the data. We consider different philosophies for achieving this goal and suggest strategies for data analysis. We illustrate our methods through three examples. The first is a Poisson regression of bird counts in which a choice is to be made between inclusion of one or both of two covariates. The second is a line transect data set for which different models yield substantially different estimates of abundance. The third is a simulated example in which truth is known. <s> BIB002 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> THE MATHEMATICS BEHIND MODEL AVERAGING <s> Model averaging is a group of methods for combining predictions from several models which have the benefit of considering model uncertainty in addition to parameter uncertainty. The aim of this paper is to introduce these methods in the context of mechanistic model development. In model averaging predictions are combined, by weighting with factors related to model performance, resulting in ensemble predictions. Bayesian Model Averaging (BMA) is model averaging in a Bayesian framework where the model weights are Posterior model probabilities (PMPs). We describe three approximation methods (AIC, BIC and Laplace) for calculating PMPs and to compare with a full Bayesian approach implemented using a Markov Chain Monte Carlo (MCMC) method (Metropolis-Hastings). We also describe a simplified BMA approach which is readily implemented, as it only requires the maximum likelihood parameter estimates and Laplace approximation of the marginal likelihoods. We illustrate the application of BMA using a mechanistic model for predicting the plant uptake of radiocaesium from contaminated soils (the 'Absalom Model'). Ten models were selected for averaging, these comprised the full Absalom model and nine reduced models each derived from the full model. To assess performance model predictions and ensemble predictions were compared using an independent data set. The PMPs estimated using the MCMC approach and the Laplace approximation were similar and strongly weighted the models with fewer parameters. The AIC- and BIC-based estimates of the PMPs were correlated but differed considerably from the Laplace and MCMC-based PMP methods. For our example the simplified BMA approach was performed as well as the full approach. Individual predictions differed among models and the prediction ensembles resulting from all the approaches captured this uncertainty. We conclude that BMA is a valuable approach, relevant to mechanistic model development, and suggest a framework for incorporating BMA into model development. <s> BIB003
In accordance with virtually all discussions of model averaging we encountered, we first focus on how model averaging reduces prediction error, here quantified as mean squared error (MSE) of a prediction b Y m of model m. As for any estimator, we can decompose this error into contributions of bias and variance: Bias refers to a systematic model error that would not change if a new data set for the same system became available, while variance refers to the expected spread of model predictions when fit with hypothetical new data sets for the same system. We can use Eq. 1 to examine the error of a weighted average e Y of the predictions of several (M) contributing models, b The motivation for the weights w m is to adjust the average such that is has improved properties over a simple average (with equal weights) or a single candidate models (all weight on one model). We can see from Eq. 1 that bias, i.e., the difference between the expectation of the averaged predictions and the truth ( e Y À y Ã ), will depend directly on the bias of the contributing models, as well as their weights (Eq. 2). The statistical model-averaging literature often assumes that individual models have no bias, and therefore tends to be less interested in its contribution BIB001 BIB002  . In contrast, for process models, reducing bias is often named as one of the main motivations for model averaging (e.g.,  BIB003  . Implicitly, the assumption here is that model biases will tend to fall on both sides of the truth, in which case they may cancel out in an average. Prediction variance (arising from n hypothetical repeated samplings) is composed of two terms, the variance of each Understanding what influences the error of model-averaged predictions Eq. 5 allows us to make a number of statements about the potential benefits of model averaging. We shall first illustrate the fundamental effects of bias, variance, and covariance using simple toy examples. In the next sections, we shall then move from this idealized examples to more realistic situations. First, when each model produces a distinct prediction, with variances substantially lower than systematic differences between models, bias dominates ( Fig. 1 top) . How useful model averaging is in this situation depends on the biases of the individual models (see also Fig. 2 top row). As model variance increases (or bias decreases), the error term is increasingly dominated by variance, and assuming covariances are low, the variance of the average (and therefore the mean error) will be smaller than the variance of the single model ( Fig. 1 bottom) . If the covariance of model predictions is low, increasing the number of models in the average will generally decrease the variance and therefore the prediction error, while the bias of the average has no general connection to the number of averaged models (Fig. 2 , right column). We thus conclude that, as bias becomes large relative to prediction variance, model averaging is less and less likely to be useful for reducing variancebut it may still be useful for reducing bias (under the condition of bidirectional bias: Fig. 2 , lower row). To understand these effects in more detail, consider the unlikely, but didactically important case that model predictions are independent, meaning that their covariance is 0 and the correlation matrix q mn of Eq. 5 becomes the identity matrix (or, equivalently, the covariance term of Eq. 4 vanishes). If we also assume both predictions have equal variances, varð b If the two models receive equal weight, we have varð e Y Þ ¼ ð2 Â 0:5 2 À 2 Â 0:5 þ 1Þvarð b Y Þ ¼ 0:5varð b Y Þ, a considerable improvement in prediction variance (and the minimum of this equation). Other weights fall between these values. In other words, model averaging can reduce prediction error because weights enter as quadratic terms in Eq. 3, rather than linearly. Indeed, BIB001 showed that for unbiased models with uncorrelated predictions, the variance in the average is never greater than the smaller of the individual predictions (making the important assumption that the weights are known, which will be discussed in section Estimating weights can thwart the benefit of model averaging). The next thing to note is that the correlation between model predictions, i.e., the matrix ðq ij Þ 2 R MÂM , substantially affects the benefit of model averaging (see also Fig. 3 and interactive tool in Data S1). In the best case, correlations between model predictions are negative or at least absent, and the second term of Eq. 5 is negative or vanishes. Under these conditions, averaging can substantially decrease the variance of the averaged prediction. As correlations between predictions increase, the covariance term contributes more and more to the overall prediction error. In the extreme case of perfectly correlated predictions of the single models, model averaging has no benefit for reducing prediction variance. The effect of correlations on the potential reduction of prediction error has an analogy in biodiversity studies, where it is called the "portfolio effect" (e.g., Thibaut and Connolly 2013). FIG. 2 . Conceptualized outcomes of model averaging. Sampling distributions of model predictions are depicted as stylized empty triangle on the see-saw (wider means less certain). Filled triangles represent the model predictions with unidirectionally bias (top row) or straddling truth (bottom row), and positive, no, or negative covariances among model predictions in columns. In the top row, grey-shaded quadrants indicate model combinations with bias in the same direction, leading to a biased average (tilted see-saw). In the bottom row, grey-shaded quadrants indicate opposite biases, which may lead to less biased averaged prediction, assuming optimal model weights were found. Changes in prediction covariance (columns) affect the uncertainty of the average, with negatively correlated predictions (right) yielding lowest uncertainty.