Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Dependencies among model predictions should be addressed <s> Abstract We consider the problem of model selection and accounting for model uncertainty in high-dimensional contingency tables, motivated by expert system applications. The approach most used currently is a stepwise strategy guided by tests based on approximate asymptotic P values leading to the selection of a single model; inference is then conditional on the selected model. The sampling properties of such a strategy are complex, and the failure to take account of model uncertainty leads to underestimation of uncertainty about quantities of interest. In principle, a panacea is provided by the standard Bayesian formalism that averages the posterior distributions of the quantity of interest under each of the models, weighted by their posterior model probabilities. Furthermore, this approach is optimal in the sense of maximizing predictive ability. But this has not been used in practice, because computing the posterior model probabilities is hard and the number of models is very large (often greater than 1... <s> BIB001 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Dependencies among model predictions should be addressed <s> In most examples of inference and prediction, the expression of uncertainty about unknown quantities y on the basis of known quantities x is based on a model M that formalizes assumptions about how x and y are related. M will typically have two parts: structural assumptions S, such as the form of the link function and the choice of error distribution in a generalized linear model, and parameters θ whose meaning is specific to a given choice of S. It is common in statistical theory and practice to acknowledge parametric uncertainty about θ given a particular assumed structure S; it is less common to acknowledge structural uncertainty about S itself. A widely used approach involves enlisting the aid of x to specify a plausible single «best» choice S* for S, and then proceeding as if S* were known to be correct. In general this approach fails to assess and propagate structural uncertainty fully and may lead to miscalibrated uncertainty assessments about y given x. When miscalibration occurs it will often result in understatement of inferential or predictive uncertainty about y, leading to inaccurate scientific summaries and overconfident decisions that do not incorporate sufficient hedging against uncertainty. In this paper I discuss a Bayesian approach to solving this problem that has long been available in principle but is only now becoming routinely feasible, by virtue of recent computational advances, and examine its implementation in examples that involve forecasting the price of oil and estimating the chance of catastrophic failure of the US space shuttle <s> BIB002 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Dependencies among model predictions should be addressed <s> Although bioclimatic modelling is often used to estimate potential impacts of likely climate changes, little has been done to assess the reliability and variability of projections. Here, using four niche-based models, two methods to derive probability values from models into presence–absence data and five climate change scenarios, I project the future potential habitats of 1350 European plant species for 2050. All 40 different projections of species turnover across Europe suggested high potential species turnover (up to 70%) in response to climate change. However variability in the potential distributional changes of species across climate scenarios was obscured by a strong variability in projections arising from alternative, yet equally justifiable, niche-based models. Therefore, projections of future species distributions and derived community descriptors cannot be reliably discussed unless model uncertainty is quantified explicitly. I propose and test an alternative way to account for modelling variability when deriving estimates of species turnover (with and without dispersal) according to a range of climate change scenarios representing various socio-economic futures. <s> BIB003 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Dependencies among model predictions should be addressed <s> Model-combining (i.e., mixing) methods have been proposed in recent years to deal with uncertainty in model selection. Even though advantages of model combining over model selection have been demonstrated in simulations and data examples, it is still unclear to a large extent when model combining should be preferred. In this work, first we propose an instability measure to capture the uncertainty of model selection in estimation, called perturbation instability in estimation (PIE), based on perturbation of the sample. We demonstrate that estimators from model selection can have large PIE values and that model combining substantially reduces the instability for such cases. Second, we propose a model combining method, adaptive regression by mixing with model screening (ARMS), and derive a theoretical property. In ARMS, a screening step is taken to narrow down the list of candidate models before combining, which not only saves computing time, but also can improve estimation accuracy. Third, we compare ARMS w... <s> BIB004 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Dependencies among model predictions should be addressed <s> Recently, penalized regression methods have attracted much attention in the statistical literature. In this article, we argue that such methods can be improved for the purposes of prediction by utilizing model averaging ideas. We propose a new algorithm that combines penalized regression with model averaging for improved prediction. We also discuss the issue of model selection versus model averaging and propose a diagnostic based on the notion of generalized degrees of freedom. The proposed methods are studied using both simulated and real data. <s> BIB005 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Dependencies among model predictions should be addressed <s> A variety of predictive models is currently used to map the spatial distribution of earth surface processes and landforms. In this study, we tested statistical consensus methods in order to improve the predictive accuracy of geomorphological models. The distributions of 12 geomorphological formations were recorded at a resolution of 25ha in a sub-arctic landscape in northern Finland. Nine environmental variables were used to predict probabilities of occurrence of the formations using eight state-of-the-art modelling techniques. The probability values of the models were combined using four different consensus methods. The accuracy of the models was calculated using spatially independent test data by the area under the curve (AUC) of a receiver-operating characteristic (ROC) plot. The mean AUC values of the geomorphological models varied between 0.711 and 0.755 based on single-model techniques, whereas the corresponding values based on consensus methods ranged from 0.752 to 0.782. The weighted average consensus method had the highest predictive performance of all methods. It improved the accuracy of 11 predictions out of 12. The results of this study suggest that the consensus methods have clear advantages over single-model predictions. The simplicity of the consensus methods makes it straightforward to implement them in predictive modelling studies in geomorphology. <s> BIB006 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Dependencies among model predictions should be addressed <s> We address the task of choosing prior weights for models that are to be used for weighted model averaging. Models that are very similar should usually be given smaller weights than models that are quite distinct. Otherwise, the importance of a model in the weighted average could be increased by augmenting the set of models with duplicates of the model or virtual duplicates of it. Similarly, the importance of a particular model feature (a certain covariate, say) could be exaggerated by including many models with that feature. Ways of forming a correlation matrix that reflects the similarity between models are suggested. Then, weighting schemes are proposed that assign prior weights to models on the basis of this matrix. The weighting schemes give smaller weights to models that are more highly correlated. Other desirable properties of a weighting scheme are identified, and we examine the extent to which these properties are held by the proposed methods. The weighting schemes are applied to real data, and prior weights, posterior weights and Bayesian model averages are determined. For these data, empirical Bayes methods were used to form the correlation matrices that yield the prior weights. Predictive variances are examined, as empirical Bayes methods can result in unrealistically small variances. <s> BIB007 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Dependencies among model predictions should be addressed <s> The aim of this study was to analyse the effects of species geographical and environmental ranges on the predictive performances of species distribution models (SDMs). We explored the usefulness of ensemble modelling approaches and tested whether species attributes influenced the outcomes of such approaches. Eight SDMs were used to model the current distribution of 35 fish species at 1110 stream sections in France. We first quantified the consensus among the resulting set of predictions for each fish species. Next, we created an average model by taking the average of the individual model predictions and tested whether the average model improved the predictive performances of single SDMs. Lastly, we described the ranges of fish species along four gradients: latitudinal, thermal, stream gradient (i.e. upstream-downstream) and elevation. After accounting for the effects of phylogenetic relatedness and species prevalence, these four species attributes were related to the observed variations in both consensus among SDMs and predictive performances by using generalized estimation equations. Our results highlight the usefulness of ensemble approaches for identifying geographical areas of agreement among predictions. Although the geographical extent of species had no effect on the performances of SDMs, we demonstrated that more consensual and accurate predictions were obtained for species with low thermal and elevation ranges, validating the hypothesis that specialist species yield models with higher accuracy than generalist ones. We emphasized that significant improvements in the accuracy of SDMs can be achieved by using an average model. Furthermore, these improvements were higher for species with smaller ranges along the four gradients studied. The geographical extent and ranges of species along environmental gradients provide promising insights into our understanding of uncertainties in species distribution modelling. <s> BIB008 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Dependencies among model predictions should be addressed <s> Aim ::: Species distribution models (SDMs) are commonly used to forecast climate change impacts. These models, however, are subject to important assumptions and limitations. By integrating two independent but complementary methods, ensemble SDMs and statistical phylogeography, we addressed key assumptions and created robust assessments of climate change impacts on species distributions while improving the conservation value of these projections. ::: ::: Location ::: North American cordillera. ::: ::: Methods ::: This approach was demonstrated using the arctic-alpine plant Rhodiola integrifolia (Crassulaceae). SDMs were fitted to current and past climates using eight models, two thresholds and one to three climate data sets. These projections were combined to create a map of stable climate (refugia) since the Last Interglacial (124,000 kya). Five biogeographic hypotheses were developed based on the configuration of refugia and tested using statistical phylogeography. Projection of SDMs into the future was contingent on agreement across approaches; future projections (to 2085) used five climate data sets and two greenhouse gas scenarios. ::: ::: Results ::: A multiple-refugia hypothesis was supported by both methods, confirming the assumption of niche conservatism in R. integrifolia and justifying the projection of SDMs onto future climates. Future projections showed substantial loss of climatically suitable habitat. Southern populations had the greatest losses, although the biogeographic scale of modelling may overpredict extinction risks in areas of topographic complexity. Past and future SDMs were assessed for novel values of climate variables; areas of novel climate were flagged as having higher uncertainty. ::: ::: Main conclusions ::: Integrating molecular approaches with spatial analyses of species distributions under global change has great potential to improve conservation decision-making. Molecular tools can support and improve current methods for understanding the vulnerability of species to climate change and provide additional data upon which to base conservation decisions, such as prioritizing the conservation of areas of high genetic diversity to build evolutionary resiliency within populations. <s> BIB009 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Dependencies among model predictions should be addressed <s> Methodological absences, i.e. when a species is not detected although it is actually present, are known to reduce the prediction accuracy of species distribution models (SDMs). To deal with this problem, we assessed whether a new iterative ensemble modelling (IEM) approach better predicts the spatial distribution of a set of 31 freshwater fish species, exhibiting a wide range of prevalence and methodological absences. Model efficiency was compared using one threshold-independent (AUC) and three threshold-dependent indicators of model predictive performance: the percentage of misclassified sites; the Kappa index; and the True Skill Statistic. We then reconstructed species assemblages from individual species predictions and compared observed assemblages to those predicted using EM and IEM using the Jaccard index. Compared to an EM approach, IEM improved model predictive performance for most difficult-to-detect species. The iterative approach outperformed EM at modelling the distribution of difficult-to-detect species, provided that presence data are representative of the niche of the species. At the assemblage level, the discrepancy between observed and IEM predicted assemblages was significantly lower than that between observed and EM predicted assemblages, showing that IEM can be used to predict the distribution of entire species assemblages. The IEM approach provides a way to consider difficult-to-detect species in species distribution models. <s> BIB010 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Dependencies among model predictions should be addressed <s> The steady upward trend in the use of model selection and Bayesian methods in ecological research has made it clear that both approaches to inference are important for modern analysis of models and data. However, in teaching Bayesian methods and in working with our research colleagues, we have noticed a general dissatisfaction with the available literature on Bayesian model selection and multimodel inference. Students and researchers new to Bayesian methods quickly find that the published advice on model selection is often preferential in its treatment of options for analysis, frequently advocating one particular method above others. The recent appearance of many articles and textbooks on Bayesian modeling has provided welcome background on relevant approaches to model selection in the Bayesian framework, but most of these are either very narrowly focused in scope or inaccessible to ecologists. Moreover, the methodological details of Bayesian model selection approaches are spread thinly throughout the literature, appearing in journals from many different fields. Our aim with this guide is to condense the large body of literature on Bayesian approaches to model selection and multimodel inference and present it specifically for quantitative ecologists as neutrally as possible. We also bring to light a few important and fundamental concepts relating directly to model selection that seem to have gone unnoticed in the ecological literature. Throughout, we provide only a minimal discussion of philosophy, preferring instead to examine the breadth of approaches as well as their practical advantages and disadvantages. This guide serves as a reference for ecologists using Bayesian methods, so that they can better understand their options and can make an informed choice that is best aligned with their goals for inference. <s> BIB011 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Dependencies among model predictions should be addressed <s> Ecological data often show temporal, spatial, hierarchical (random effects), or phylogenetic structure. Modern statistical approaches are increasingly accounting for such dependencies. However, when performing cross-validation, these structures are regularly ignored, resulting in serious underestimation of predictive error. One cause for the poor performance of uncorrected (random) cross-validation, noted often by modellers, are dependence structures in the data that persist as dependence structures in model residuals, violating the assumption of independence. Even more concerning, because often overlooked, is that structured data also provides ample opportunity for overfitting with non-causal predictors. This problem can persist even if remedies such as autoregressive models, generalized least squares, or mixed models are used. Block cross-validation, where data are split strategically rather than randomly, can address these issues. However, the blocking strategy must be carefully considered. Blocking in space, time, random effects or phylogenetic distance, while accounting for dependencies in the data, may also unwittingly induce extrapolations by restricting the ranges or combinations of predictor variables available for model training, thus overestimating interpolation errors. On the other hand, deliberate blocking in predictor space may also improve error estimates when extrapolation is the modelling goal. Here, we review the ecological literature on non-random and blocked cross-validation approaches. We also provide a series of simulations and case studies, in which we show that, for all instances tested, block cross-validation is nearly universally more appropriate than random cross-validation if the goal is predicting to new data or predictor space, or for selecting causal predictors. We recommend that block cross-validation be used wherever dependence structures exist in a dataset, even if no correlation structure is visible in the fitted model residuals, or if the fitted models account for such correlations. <s> BIB012
Statistical models, which aim to describe the data to which they are fitted, will often have correlated parameters and fits; process models may overlap in modelled processes. Having highly similar models in the model set will inflate the cumulative weight given to them (as illustrated in Appendix S1.6). One way to handle inflation of weights by highly related models is to assign prior model probabilities in a Bayesian framework. Another approach would be to pre-select models of different types (see next point). Alternatively, the cossquare scheme of BIB007 uses the correlation matrix of model projections to appropriately change weights of correlated models. Of the weighting schemes considered here, it is the only approach doing so, but it should be noted that the performance of this approach in our case study was rather poor (Fig. 6, Tables 2 and 3) . Validation-based weighting or validation-based pre-selection of models BIB001 , BIB002 ,  , and more recently BIB004 and BIB005 , have argued that only "good" models should be averaged. Different ways of combining model averaging with a model screening step have been proposed  BIB004 BIB005 ), in which model selection precedes averaging (pre-selection). This will happen implicitly, and in a single step, if any of the model weight algorithms discussed above attributes a weight of effectively zero to a model, as happened in case study 2. How prevalent this effect is in real world studies is unclear, as weights are rarely reported. In contrast, some studies select models after the predictions are made (e.g., BIB003 BIB009 ). These studies have averaged either models that predict in the same direction (along the "consensus axis"; BIB008 , or the best 50% in the set BIB006 ), or however many models one should combine to minimize prediction error. Such approaches necessitate addressing the challenge of using data twice BIB010 . Post-selection reduces the ability of "dissenting voices" (i.e., less correlated predictions) to reduce prediction error and instead reinforce the trend of emphasizing the model type most represented in the set. As a consequence, their uncertainty estimation will be overly optimistic. We do not advocate their use. We suggest to employ validation-based methods of model averaging rather than relying on model-based estimates of error. That is, we recommend (leave-one out) cross-validation and stacking rather than AIC (in line with recommendations of BIB011 . Using (semi-) independent test data gives us some capacity to estimate predictive bias. In such a setting, it may be less relevant whether models are pre-selected by validation-based estimates of error and then averaged with equal weights or weighted by validation-based estimates of error without pre-selection. For this to work, however, it is crucial that (cross)-validation strategies are designed to ensure independence of the validation data, which is a non-trivial problem in many practical ecological applications BIB012 ).