Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Equal weights <s> Weed risk assessment has become an accepted methodology for examining the likelihood and consequence of a plant species becoming invasive outside of its native range. Weed risk assessment draws upon biological and ecological information to estimate the likelihood and magnitude of the threats posed by introducing non-indigenous plants. In geographical terms, this has traditionally been understood as within a new country following importation of plant material. However, recent risk assessment development has focused more specifically on intracountry risk posed by already-present invasive plants and is referred to as post-border weed risk management. This form of assessment calls for fine-scale predictions of invasive species habitat suitability. This study applies some of the more popular and widely available habitat prediction models that represent a variety of different statistical approaches (linear regression, logistic regression, Bayesian probability, Classification and Regression Trees, Genetic Algorithm for Rule-set Production) to a single invasive plant, the vertebrate-dispersed, fleshy fruited European olive (Olea europaea L.) in southern Australia. The relationships between the dependant (O. europaea distribution) and independent (soil and climate) variables are used in the models to produce predictive maps for each model. Accuracy was calculated for each model output as well as a combined surface to examine whether recent calls for ensemble modelling of distributions produces improved predictions. Overall, the combined prediction demonstrated superior accuracy compared to any individual model outputs. The combined outputs can be likened to mapped gradations of predicted habitat suitability. The type of output produced in this study should form a critical component of post-border weed risk management but more importantly, the methodology will add to this important discipline. <s> BIB001 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Equal weights <s> Recent coordinated efforts, in which numerous general circulation climate models have been run for a common set of experiments, have produced large datasets of projections of future climate for various scenarios. Those multimodel ensembles sample initial conditions, parameters, and structural uncertainties in the model design, and they have prompted a variety of approaches to quantifying uncertainty in future climate change. International climate change assessments also rely heavily on these models. These assessments often provide equal-weighted averages as best-guess results, assuming that individual model biases will at least partly cancel and that a model average prediction is more likely to be correct than a prediction from a single model based on the result that a multimodel average of present-day climate generally outperforms any individual model. This study outlines the motivation for using multimodel ensembles and discusses various challenges in interpreting them. Among these challenges are that the number of models in these ensembles is usually small, their distribution in the model or parameter space is unclear, and that extreme behavior is often not sampled. Model skill in simulating present-day climate conditions is shown to relate only weakly to the magnitude of predicted change. It is thus unclear by how much the confidence in future projections should increase based on improvements in simulating present-day conditions, a reduction of intermodel spread, or a larger number of models. Averaging model output may further lead to a loss of signal— for example, for precipitation change where the predicted changes are spatially heterogeneous, such that the true expected change is very likely to be larger than suggested by a model average. Last, there is little agreement on metrics to separate ‘‘good’’ and ‘‘bad’’ models, and there is concern that model development, evaluation, and posterior weighting or ranking are all using the same datasets. While the multimodel average appears to still be useful in some situations, these results show that more quantitative methods to evaluate model performance are critical to maximize the value of climate change projections from global models. <s> BIB002 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Equal weights <s> A variety of predictive models is currently used to map the spatial distribution of earth surface processes and landforms. In this study, we tested statistical consensus methods in order to improve the predictive accuracy of geomorphological models. The distributions of 12 geomorphological formations were recorded at a resolution of 25ha in a sub-arctic landscape in northern Finland. Nine environmental variables were used to predict probabilities of occurrence of the formations using eight state-of-the-art modelling techniques. The probability values of the models were combined using four different consensus methods. The accuracy of the models was calculated using spatially independent test data by the area under the curve (AUC) of a receiver-operating characteristic (ROC) plot. The mean AUC values of the geomorphological models varied between 0.711 and 0.755 based on single-model techniques, whereas the corresponding values based on consensus methods ranged from 0.752 to 0.782. The weighted average consensus method had the highest predictive performance of all methods. It improved the accuracy of 11 predictions out of 12. The results of this study suggest that the consensus methods have clear advantages over single-model predictions. The simplicity of the consensus methods makes it straightforward to implement them in predictive modelling studies in geomorphology. <s> BIB003 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Equal weights <s> Conservation planners often wish to predict how species distributions will change in response to environmental changes. Species distribution models (SDMs) are the primary tool for making such predictions. Many methods are widely used; however, they all make simplifying assumptions, and predictions can therefore be subject to high uncertainty. With global change well underway, field records of observed range shifts are increasingly being used for testing SDM transferability. We used an unprecedented distribution dataset documenting recent range changes of British vascular plants, birds, and butterflies to test whether correlative SDMs based on climate change provide useful approximations of potential distribution shifts. We modelled past species distributions from climate using nine single techniques and a consensus approach, and projected the geographical extent of these models to a more recent time period based on climate change; we then compared model predictions with recent observed distributions in order to estimate the temporal transferability and prediction accuracy of our models. We also evaluated the relative effect of methodological and taxonomic variation on the performance of SDMs. Models showed good transferability in time when assessed using widespread metrics of accuracy. However, models had low accuracy to predict where occupancy status changed between time periods, especially for declining species. Model performance varied greatly among species within major taxa, but there was also considerable variation among modelling frameworks. Past climatic associations of British species distributions retain a high explanatory power when transferred to recent time--due to their accuracy to predict large areas retained by species--but fail to capture relevant predictors of change. We strongly emphasize the need for caution when using SDMs to predict shifts in species distributions: high explanatory power on temporally-independent records--as assessed using widespread metrics--need not indicate a model's ability to predict the future. <s> BIB004 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Equal weights <s> Species distributions models (SDMs) are commonly used to assess potential species’ range shifts or extinction risk under climate change. It has been suggested that the use of ensemble forecasts, where a variety of model algorithms are used to generate consensus predictions, are preferred to individual SDMs by avoiding bias or prediction error inherent in a single modeling approach. Whereas several studies have assessed the performance of ensemble predictions using cross-validation or data-partitioning approaches, few studies have assessed the predictive accuracy of ensemble forecasts under climate change by using temporally independent model validation data. We used five SDM approaches to develop consensus forecasts of distributions of 145 vascular plant species from California in the 1930s and tested their projections against current distributions, a span of approximately 75 years. When evaluated with a portion of the model training data, consensus forecasts were highly accurate with an average AUC value of 0.97. False positive and false negative error rates were also low, exhibiting similar performance to random forest models. However, when evaluated with temporally independent data, the accuracy of consensus forecasts was similar to that of generalized linear and generalized additive models, with an average AUC value of 0.83. Our results suggest that the high levels of predictive accuracy exhibited by consensus forecasts when using data partitioning approaches may not reflect their performance when predicting temporally independent data. We contend that consensus forecasts may not represent the best approach for predicting species distributions under future climatic change, as they may not provide superior predictive accuracy in novel temporal domains compared to traditional modeling approaches that more readily lend themselves to ecological interpretation of model structure. <s> BIB005 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Equal weights <s> We summarize the literature on the effectiveness of combining forecasts by assessing the conditions under which combining is most valuable. Using data on the six US presidential elections from 1992 to 2012, we report the reductions in error obtained by averaging forecasts within and across four election forecasting methods: poll projections, expert judgment, quantitative models, and the Iowa Electronic Markets. Across the six elections, the resulting combined forecasts were more accurate than any individual component method, on average. The gains in accuracy from combining increased with the numbers of forecasts used, especially when these forecasts were based on different methods and different data, and in situations involving high levels of uncertainty. Such combining yielded error reductions of between 16% and 59%, compared to the average errors of the individual forecasts. This improvement is substantially greater than the 12% reduction in error that had been reported previously for combining forecasts. <s> BIB006 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Equal weights <s> AbstractIn fields such as climate science, it is common to compile an ensemble of different simulators for the same underlying process. It is a striking observation that the ensemble mean often outperforms at least half of the ensemble members in mean squared error (measured with respect to observations). In fact, as demonstrated in the most recent IPCC report, the ensemble mean often outperforms all or almost all of the ensemble members across a range of climate variables. This paper shows that these could be mathematical results based on convexity and averaging but with implications for the properties of the current generation of climate simulators. <s> BIB007
Last, we discuss the most trivial weighting scheme: in many fields of science (climate modelling, economics, political sciences), model averaging proceeds with giving the structurally different models equal weight, i.e., 1/M (Johnson and Bowler 2009, BIB002 BIB006 BIB007 . In ecology, studies analyzing species distributions reported equal weights to be a very good choice when assessed using cross-validation BIB001 BIB003 BIB004 , but no better than the single models on validation with independent data BIB005 . Equal weights may serve as a FIG. 5. A simple model-based model combination (MBMC) example. (A) Three models (solid grey lines: constant, linear, and quadratic) fitted separately to a data set (points, following the thin black line). Using a linear model (with quadratic terms: red) to combine the three models' its may improve it, even more so than the full model (green), and with narrower conidence intervals. Doted lines indicate the weight that each model receives at each point in the linear model. Such MBMC did not necessarily improve fit, as Random Forest-based model combinations showed (blue). (B) Using fivefold cross-validation around the entire workflow shows that the linear supra-model (Supra-LM) indeed improved prediction (decreased root mean squared prediction error), while the Random Forest-supra-model (Supra-rF) did not. The full model (as reference) comprised all terms present in Supra-LM, but was fitted directly. Boxes are 50% quantiles, and whiskers are min/max values (unless those exceed the 50% quantiles by 1.5 the interquartile range); the median is indicated by the horizontal black line.