Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> 488 <s> Two separate sets of forecasts of airline passenger data have been combined to form a composite set of forecasts. The main conclusion is that the composite set of forecasts can yield lower mean-square error than either of the original forecasts. Past errors of each of the original forecasts are used to determine the weights to attach to these two original forecasts in forming the combined forecasts, and different methods of deriving these weights are examined. <s> BIB001 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> 488 <s> Abstract Considerable literature has accumulated over the years regarding the combination of forecasts. The primary conclusion of this line of research is that forecast accuracy can be substantially improved through the combination of multiple individual forecasts. Furthermore, simple combination methods often work reasonably well relative to more complex combinations. This paper provides a review and annotated bibliography of that literature, including contributions from the forecasting, psychology, statistics, and management science literatures. The objectives are to provide a guide to the literature for students and researchers and to help researchers locate contributions in specific areas, both theoretical and applied. Suggestions for future research directions include (1) examination of simple combining approaches to determine reasons for their robustness, (2) development of alternative uses of multiple forecasts in order to make better use of the information they contain, (3) use of combined forecasts as benchmarks for forecast evaluation, and (4) study of subjective combination procedures. Finally, combining forecasts should become part of the mainstream of forecasting practice. In order to achieve this, practitioners should be encouraged to combine forecasts, and software to produce combined forecasts easily should be made available. <s> BIB002 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> 488 <s> There has been increasing interest recently in model averaging within the frequentist paradigm. The main benefit of model averaging over model selection is that it incorporates rather than ignores the uncertainty inherent in the model selection process. One of the most important, yet challenging, aspects of model averaging is how to optimally combine estimates from different models. In this work, we suggest a procedure of weight choice for frequentist model average estimators that exhibits optimality properties with respect to the estimator’s mean squared error (MSE). As a basis for demonstrating our idea, we consider averaging over a sequence of linear regression models. Building on this base, we develop a model weighting mechanism that involves minimizing the trace of an unbiased estimator of the model average estimator’s MSE. We further obtain results that reflect the finite sample as well as asymptotic optimality of the proposed mechanism. A Monte Carlo study based on simulated and real data evaluates... <s> BIB003 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> 488 <s> We summarize the literature on the effectiveness of combining forecasts by assessing the conditions under which combining is most valuable. Using data on the six US presidential elections from 1992 to 2012, we report the reductions in error obtained by averaging forecasts within and across four election forecasting methods: poll projections, expert judgment, quantitative models, and the Iowa Electronic Markets. Across the six elections, the resulting combined forecasts were more accurate than any individual component method, on average. The gains in accuracy from combining increased with the numbers of forecasts used, especially when these forecasts were based on different methods and different data, and in situations involving high levels of uncertainty. Such combining yielded error reductions of between 16% and 59%, compared to the average errors of the individual forecasts. This improvement is substantially greater than the 12% reduction in error that had been reported previously for combining forecasts. <s> BIB004 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> 488 <s> This paper offers a theoretical explanation for the stylized fact that forecast combinations with estimated optimal weights often perform poorly in applications. The properties of the forecast combination are typically derived under the assumption that the weights are fixed, while in practice they need to be estimated. If the fact that the weights are random rather than fixed is taken into account during the optimality derivation, then the forecast combination will be biased (even when the original forecasts are unbiased), and its variance will be larger than in the fixed-weight case. In particular, there is no guarantee that the ‘optimal’ forecast combination will be better than the equal-weight case, or even improve on the original forecasts. We provide the underlying theory, some special cases, and a numerical illustration. <s> BIB005
DORMANN Vol. 88, No. 4 It states that the fluctuation in biomass of a community is less than the fluctuations of biomass of its members, because the species respond to the environment differently. This asynchrony in response is analogous to negative covariance in community members' biomass, buffering the sum of their biomasses. This point also provides some important insights about why machine learning methods, which often average a large number of bad models, can work so well. When averaging poor models, e.g., trees in a Random Forest, covariance is negligible, but the variance of each model prediction is high. Because w m becomes very small with hundreds of models (approximately 1/M), the variance of many averaged poor models (with similar variance) tends to be low: varð where the second term disappears due to lack of correlations among predictions. We may speculate that poor models typically also exhibit substantial but bidirectional bias, which again would be reduced by averaging. Putting bias, variance and correlation together ( Fig. 2) , we note that model averaging will deliver smaller prediction error when bias is bidirectional (i.e., model predictions overand underestimate the true value: bottom row of Fig. 2 ) and predictions are negatively correlated ( Fig. 2 bottom right) . Uni-directional bias will remain problematic (top row of Fig. 2 ), irrespective of covariances among predictions. Thus, for a given set of weights, the prediction error of model-averaged predictions depends on three things: the bias of the model average, as emerging from the bias of the individual models, the prediction variances of the individual models, and the covariance of those predictions. Estimating weights can thwart the benefit of model averaging So far, we have assumed that weights have fixed values, and thus there is no uncertainty about them. Yet, the aim of optimizing predictive performance suggests that weights need to be estimated from the data. But estimation brings associated uncertainty with it, and this has implications for the actual benefits of model averaging: estimated "optimal" weights will be suboptimal (Nguefack-Tsague 2014). With such an error, even for only mildly correlated predictions, the averaged prediction will have more variance, and possibly bias, than if the (unknown) truly optimal weights were used BIB005 . It may, in fact, often be no better than one obtained using arbitrary weights, e.g., equal weights BIB002 BIB004 . The "simple theoretical explanation" provided by BIB005 demonstrates that estimating weights introduces additional variance into the prediction. As a consequence, the predictions averaged with estimated weights may be worse than that of a single model (in contrast to the assertion of BIB001 Granger 1969, see Claeskens et al. 2016 for an example). Apart from the error of the estimate, a further open problem is to obtain a good estimator for the optimal weight in the first place. Currently no closed solution is available, not even for linear models BIB003 ). Neither Bayesian nor information-theoretical model weights are designed to minimize prediction error, and their weights will in general not be optimal for that purpose. Some tactical approaches estimate model weights explicitly to minimize prediction error on hold-out data (in particular jackknife model averaging and stacking; see Tactical approaches to computing FIG. 3. When averaging is optimal, in the simplest case of two models that make correlated Gaussian predictions. The models are here described by their biases (b 1 , b 2 , not shown), their standard deviations (r 1 , r 2 ), and by the correlation (q) between them. Each panel shows the regions in the ðr 1 ; qÞ plane where model 1 is best (blue shading and contour line), model 2 is best (orange shading and contour line), and where the optimal average is best (color gradient between blue and orange). Top row represents the case where weights, w, are known (i.e., without error: r w ¼ 0), while the second row represents exactly the same settings, but with estimated weights (with uncertainty r w ¼ 0:2). Notice that when w is estimated with uncertainty, the contours marking the transition between each single model and the average move into the washed-out colours, i.e., deviate from the fixed w situation in the upper panels. These curves now represent a level set at the values w Ã 1 ¼ 1 À r w (blue curve) and w Ã 2 ¼ r w (orange curve). As a consequence, the area where model averaging with estimated weights is superior to the better single model decreases substantially relative to the fixed w case, and disappears completely for r w ! 0:5. Formal derivations for the contours and the critical weights is given in Appendix S1.2, the interactive tool itself in Data S1. Biases are set to b 1 ¼ 3 and b 2 ¼ 2.