Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Model averaging (typically) reduces prediction errors <s> Central to statistical theory and application is statistical modeling, which typically involves choosing a single model or combining a number of models of different sizes and from different sources. Whereas model selection seeks a single best modeling procedure, model combination combines the strength of different modeling procedures. In this article we look at several key issues and argue that model assessment is the key to model selection and combination. Most important, we introduce a general technique of optimal model assessment based on data perturbation, thus yielding optimal selection, in particular model selection and combination. From a frequentist perspective, we advocate model combination over a selected subset of modeling procedures, because it controls bias while reducing variability, hence yielding better performance in terms of the accuracy of estimation and prediction. To realize the potential of model combination, we develop methodologies for determining the optimal tuning parameter, such... <s> BIB001 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Model averaging (typically) reduces prediction errors <s> The alleviation of food-borne diseases caused by microbial pathogen remains a great concern in order to ensure the well-being of the general public. The relation between the ingested dose of organisms and the associated infection risk can be studied using dose-response models. Traditionally, a model selected according to a goodness-of-fit criterion has been used for making inferences. In this article, we propose a modified set of fractional polynomials as competitive dose-response models in risk assessment. The article not only shows instances where it is not obvious to single out one best model but also illustrates that model averaging can best circumvent this dilemma. The set of candidate models is chosen based on biological plausibility and rationale and the risk at a dose common to all these models estimated using the selected models and by averaging over all models using Akaike's weights. In addition to including parameter estimation inaccuracy, like in the case of a single selected model, model averaging accounts for the uncertainty arising from other competitive models. This leads to a better and more honest estimation of standard errors and construction of confidence intervals for risk estimates. The approach is illustrated for risk estimation at low dose levels based on Salmonella typhi and Campylobacter jejuni data sets in humans. Simulation studies indicate that model averaging has reduced bias, better precision, and also attains coverage probabilities that are closer to the 95% nominal level compared to best-fitting models according to Akaike information criterion. <s> BIB002 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Model averaging (typically) reduces prediction errors <s> A variety of predictive models is currently used to map the spatial distribution of earth surface processes and landforms. In this study, we tested statistical consensus methods in order to improve the predictive accuracy of geomorphological models. The distributions of 12 geomorphological formations were recorded at a resolution of 25ha in a sub-arctic landscape in northern Finland. Nine environmental variables were used to predict probabilities of occurrence of the formations using eight state-of-the-art modelling techniques. The probability values of the models were combined using four different consensus methods. The accuracy of the models was calculated using spatially independent test data by the area under the curve (AUC) of a receiver-operating characteristic (ROC) plot. The mean AUC values of the geomorphological models varied between 0.711 and 0.755 based on single-model techniques, whereas the corresponding values based on consensus methods ranged from 0.752 to 0.782. The weighted average consensus method had the highest predictive performance of all methods. It improved the accuracy of 11 predictions out of 12. The results of this study suggest that the consensus methods have clear advantages over single-model predictions. The simplicity of the consensus methods makes it straightforward to implement them in predictive modelling studies in geomorphology. <s> BIB003 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Model averaging (typically) reduces prediction errors <s> Abstract. In this paper we investigate some basic properties of the multi-model ensemble systems, which can be deduced from a general characteristic of statistical distributions of the ensemble members with the help of mathematical tools. In particular we show how to find optimal linear combination of model results, which minimizes the mean square error both in the case of uncorrelated and correlated models. By proving basic estimations we try to deduce general properties describing multi-model ensemble systems. We show also how mathematical formalism can be used for investigation of the characteristics of such systems. <s> BIB004 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Model averaging (typically) reduces prediction errors <s> We extend ensemble Bayesian model averaging (EBMA) for application to binary outcomes and illustrate EBMA’s ability to aid scholars in the social sciences to make more accurate forecasts of future events. In essence, EBMA improves prediction by pooling information from multiple forecast models to generate ensemble predictions similar to a weighted average of component forecasts. The weight assigned to each forecast is calibrated via its performance in some training period. The aim is not to choose some “best” model, but rather to incorporate the insights and knowledge implicit in various forecasting efforts via statistical postprocessing. After presenting the method, we show that EBMA increases the accuracy of out-of-sample forecasts relative to component models in three applied examples: predicting the occurrence of insurgencies around the Pacific Rim, forecasting vote shares in U.S. presidential elections, and predicting the votes of U.S. Supreme Court justices. <s> BIB005 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Model averaging (typically) reduces prediction errors <s> Abstract The ability to map vegetation and in particular individual trees is a key component in forest management and long-term forest monitoring. Here we present a novel approach for mapping individual tree species based on ensemble modeling, i.e. combining the projections of several modeling techniques in order to reduce uncertainty. Using statistical modeling in conjunction with high-resolution aerial imagery (50 cm spatial resolution) and topo-climatic variables (5 m spatial resolution), we map the distributions of six major tree species (3 broadleaf and 3 conifers) in a study area of North-Eastern Switzerland. We also compare the relative predictive power of both topo-climatic and remote-sensing variables for mapping the spatial tree patterns and assess the importance of calibration data quality on model performance. We evaluate our projections using cross-validation as well as with independent data. Overall, the evaluations that we obtain for our vegetation maps are in line with, or higher than, those in similar studies. Depending on the considered tree species, 47.8–85.6% of our samples were correctly predicted, and we obtain an overall CCR (correct classification rate) of 0.72 and a Cohen’s kappa of 0.65. Comparing the predictive power of the different modeling techniques, we find that ensemble modeling (i.e. combining the projections of different individual modeling techniques) generally performs better than individual modeling techniques. <s> BIB006 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Model averaging (typically) reduces prediction errors <s> Linear mixed-effects models are a powerful tool for the analysis of longitudinal data. The aim of this paper is to study model averaging for linear mixed-effects models. The asymptotic distribution of the frequentist model average estimator is derived, and a confidence interval procedure with an actual coverage probability that tends to the nominal level in large samples is developed. The two confidence intervals based on the model averaging and based on the full model are shown to be asymptotically equivalent. A simulation study shows good finite sample performance of the model average estimators. <s> BIB007 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Model averaging (typically) reduces prediction errors <s> The turbulence closure model is the dominant source of error in most Reynolds-Averaged Navier-Stokes simulations, yet no reliable estimators for this error component currently exist. Here we develop a stochastic, a posteriori error estimate, calibrated to specific classes of flow. It is based on variability in model closure coefficients across multiple flow scenarios, for multiple closure models. The variability is estimated using Bayesian calibration against experimental data for each scenario, and Bayesian Model-Scenario Averaging (BMSA) is used to collate the resulting posteriors, to obtain a stochastic estimate of a Quantity of Interest (QoI) in an unmeasured (prediction) scenario. The scenario probabilities in BMSA are chosen using a sensor which automatically weights those scenarios in the calibration set which are similar to the prediction scenario. The methodology is applied to the class of turbulent boundary-layers subject to various pressure gradients. For all considered prediction scenarios the standard-deviation of the stochastic estimate is consistent with the measurement ground truth. Furthermore, the mean of the estimate is more consistently accurate than the individual model predictions. <s> BIB008 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Model averaging (typically) reduces prediction errors <s> A global trend of increasing health hazards associated with proliferation of toxin-producing cyanobacteria makes the ability to project phytoplankton dynamics of paramount importance. Whilst ensemble (multi-)modelling approaches have been used for a number of years to improve the robustness of weather forecasts this approach has until now never been adopted for ecosystem modelling. We show that the average simulated phytoplankton biomass derived from three different aquatic ecosystem models is generally superior to any of the three individual models in describing observed phytoplankton biomass in a typical temperate lake ecosystem, and we simulate a series of climate change projections. While this is the first multi-model ensemble approach applied for some of the most complex aquatic ecosystem models available, we consider it sets a precedent for what will become commonplace methodology in the future, as it enables increased robustness of model projections, and scenario uncertainty estimation due to differences in model structures. <s> BIB009 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Model averaging (typically) reduces prediction errors <s> Ecological data often show temporal, spatial, hierarchical (random effects), or phylogenetic structure. Modern statistical approaches are increasingly accounting for such dependencies. However, when performing cross-validation, these structures are regularly ignored, resulting in serious underestimation of predictive error. One cause for the poor performance of uncorrected (random) cross-validation, noted often by modellers, are dependence structures in the data that persist as dependence structures in model residuals, violating the assumption of independence. Even more concerning, because often overlooked, is that structured data also provides ample opportunity for overfitting with non-causal predictors. This problem can persist even if remedies such as autoregressive models, generalized least squares, or mixed models are used. Block cross-validation, where data are split strategically rather than randomly, can address these issues. However, the blocking strategy must be carefully considered. Blocking in space, time, random effects or phylogenetic distance, while accounting for dependencies in the data, may also unwittingly induce extrapolations by restricting the ranges or combinations of predictor variables available for model training, thus overestimating interpolation errors. On the other hand, deliberate blocking in predictor space may also improve error estimates when extrapolation is the modelling goal. Here, we review the ecological literature on non-random and blocked cross-validation approaches. We also provide a series of simulations and case studies, in which we show that, for all instances tested, block cross-validation is nearly universally more appropriate than random cross-validation if the goal is predicting to new data or predictor space, or for selecting causal predictors. We recommend that block cross-validation be used wherever dependence structures exist in a dataset, even if no correlation structure is visible in the fitted model residuals, or if the fitted models account for such correlations. <s> BIB010
To complement these theoretical considerations, we examined 180 studies (a random draws from the results of a systematic literature search: see Appendix S1.7) regarding reported benefits from model averaging. The majority of studies we encountred used an empirical approach to assess predictive performance, i.e., forecasting, hindcasting, or cross-validation to observed data (e.g., BIB002 BIB003 ,b, Grenouillet et al. 2010 BIB005 BIB006  BIB008 BIB009 . Model averaging typically yielded lower prediction errors than the individual contributing models. Most of these studies used test data sets to estimate predictive success, and rely critically on the assumption of independence between test and training data sets BIB010 . Few studies used simulated data to examine the performance of model averaging under specific conditions (e.g., small sample size, model structure uncertainty, missing data; Yuan 2009, Schomaker 2012) , and even fewer employ analytical mathematics BIB001 BIB004 BIB007 .