Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Quantifying uncertainty of model-averaged predictions <s> We argue that model selection uncertainty should be fully incorporated into statistical inference whenever estimation is sensitive to model choice and that choice is made with reference to the data. We consider different philosophies for achieving this goal and suggest strategies for data analysis. We illustrate our methods through three examples. The first is a Poisson regression of bird counts in which a choice is to be made between inclusion of one or both of two covariates. The second is a line transect data set for which different models yield substantially different estimates of abundance. The third is a simulated example in which truth is known. <s> BIB001 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Quantifying uncertainty of model-averaged predictions <s> The alleviation of food-borne diseases caused by microbial pathogen remains a great concern in order to ensure the well-being of the general public. The relation between the ingested dose of organisms and the associated infection risk can be studied using dose-response models. Traditionally, a model selected according to a goodness-of-fit criterion has been used for making inferences. In this article, we propose a modified set of fractional polynomials as competitive dose-response models in risk assessment. The article not only shows instances where it is not obvious to single out one best model but also illustrates that model averaging can best circumvent this dilemma. The set of candidate models is chosen based on biological plausibility and rationale and the risk at a dose common to all these models estimated using the selected models and by averaging over all models using Akaike's weights. In addition to including parameter estimation inaccuracy, like in the case of a single selected model, model averaging accounts for the uncertainty arising from other competitive models. This leads to a better and more honest estimation of standard errors and construction of confidence intervals for risk estimates. The approach is illustrated for risk estimation at low dose levels based on Salmonella typhi and Campylobacter jejuni data sets in humans. Simulation studies indicate that model averaging has reduced bias, better precision, and also attains coverage probabilities that are closer to the 95% nominal level compared to best-fitting models according to Akaike information criterion. <s> BIB002 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Quantifying uncertainty of model-averaged predictions <s> We consider the coverage rate of model-averaged confidence intervals for the treatment means in a factorial experiment, when we use a normal linear model in the analysis. Model-averaging provides a useful compromise between using the full model (containing all main effects and interactions) and a "best model" obtained by some model-selection process. Use of the full model guarantees perfect coverage, whereas use of a best model is known to lead to narrow intervals with poor coverage. Model-averaging allows us to achieve good coverage using intervals that are also narrower than those from the full model. We compare four information criteria that might be used for model-averaging in this setting: AIC, AICc, and BIC. In this setting, if the full model is "truth", all the criteria will have perfect coverage rates asymptotically. We use simulation to assess the coverage rates and interval widths likely to be achieved by a confidence interval with a nominal coverage of 95%. Our results suggest that AIC performs best in terms of coverage rate; across a wide range of scenarios and replication levels, it consistently provides coverage rates within 1.5% points of the nominal level, while also leading to reductions in interval-width of up to 30%, compared to the full model. AICc performed worst overall, with a coverage rate that was up to 5.2% points too low. We recommend that model-averaging become standard practise when summarising the results of a factorial experiment in terms of the treatment means, and that AIC be used to perform the model-averaging. <s> BIB003 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Quantifying uncertainty of model-averaged predictions <s> There has been increasing interest recently in model averaging within the frequentist paradigm. The main benefit of model averaging over model selection is that it incorporates rather than ignores the uncertainty inherent in the model selection process. One of the most important, yet challenging, aspects of model averaging is how to optimally combine estimates from different models. In this work, we suggest a procedure of weight choice for frequentist model average estimators that exhibits optimality properties with respect to the estimator’s mean squared error (MSE). As a basis for demonstrating our idea, we consider averaging over a sequence of linear regression models. Building on this base, we develop a model weighting mechanism that involves minimizing the trace of an unbiased estimator of the model average estimator’s MSE. We further obtain results that reflect the finite sample as well as asymptotic optimality of the proposed mechanism. A Monte Carlo study based on simulated and real data evaluates... <s> BIB004 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Quantifying uncertainty of model-averaged predictions <s> Model-averaging is commonly used as a means of allowing for model uncertainty in parameter estimation. In the frequentist framework, a model-averaged estimate of a parameter is the weighted mean of the estimates from each of the candidate models, the weights typically being chosen using an information criterion. Current methods for calculating a model-averaged confidence interval assume approximate normality of the model-averaged estimate, i.e., they are Wald intervals. As in the single-model setting, we might improve the coverage performance of this interval by a one-to-one transformation of the parameter, obtaining a Wald interval, and then back-transforming the endpoints. However, a transformation that works in the single-model setting may not when model-averaging, due to the weighting and the need to estimate the weights. In the single-model setting, a natural alternative is to use a profile likelihood interval, which generally provides better coverage than a Wald interval. We propose a method for model-averaging a set of single-model profile likelihood intervals, making use of the link between profile likelihood intervals and Bayesian credible intervals. We illustrate its use in an example involving negative binomial regression, and perform two simulation studies to compare its coverage properties with the existing Wald intervals. <s> BIB005 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Quantifying uncertainty of model-averaged predictions <s> Abstract Model specification remains challenging in spectroscopy of plant biochemistry, as exemplified by the availability of various spectral indices or band combinations for estimating the same biochemical. This lack of consensus in model choice across applications argues for a paradigm shift in hyperspectral methods to address model uncertainty and misspecification. We demonstrated one such method using Bayesian model averaging (BMA), which performs variable/band selection and quantifies the relative merits of many candidate models to synthesize a weighted average model with improved predictive performances. The utility of BMA was examined using a portfolio of 27 foliage spectral–chemical datasets representing over 80 species across the globe to estimate multiple biochemical properties, including nitrogen, hydrogen, carbon, cellulose, lignin, chlorophyll (a or b), carotenoid, polar and nonpolar extractives, leaf mass per area, and equivalent water thickness. We also compared BMA with partial least squares (PLS) and stepwise multiple regression (SMR). Results showed that all the biochemicals except carotenoid were accurately estimated from hyerspectral data with R 2 values > 0.80. Compared to PLS and SMR, BMA substantially reduced overfitting and enhanced model generalization; BMA also yielded error estimation better indicative of true uncertainties in predictions, when evaluated using a statistic called “prediction interval coverage probability”. The relative band importance, which was quantified by band selection probability, differed markedly between BMA and SMR, cautioning the use of SMR for band selection. Computationally, the model calibration with datasets of moderate sizes (> 100) was faster for BMA via a hybrid reversible-jump Monte Carlo Markov Chain sampler than for PLS via literal optimization of a cross-validation criterion. Our BMA scheme also provides a generic hierarchical Bayesian framework to assimilate prior knowledge of diverse forms, as illustrated by its use to account for nonlinearity in spectral–chemical relationships. We emphasize that BMA is a competitive, paradigm-shifting alternative to conventional statistical methods and it will find wide use as the virtue of Bayesian inference is increasingly appreciated by the remote sensing community. <s> BIB006 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Quantifying uncertainty of model-averaged predictions <s> Abstract Forest management requires prediction of forest growth, but there is no general agreement about which models best predict growth, how to quantify model parameters, and how to assess the uncertainty of model predictions. In this paper, we show how Bayesian calibration (BC), Bayesian model comparison (BMC) and Bayesian model averaging (BMA) can help address these issues. We used six models, ranging from simple parameter-sparse models to complex process-based models: 3PG, 4C, ANAFORE, BASFOR, BRIDGING and FORMIND. For each model, the initial degree of uncertainty about parameter values was expressed in a prior probability distribution. Inventory data for Scots pine on tree height and diameter, with estimates of measurement uncertainty, were assembled for twelve sites, from four countries: Austria, Belgium, Estonia and Finland. From each country, we used data from two sites of the National Forest Inventories (NFIs), and one Permanent Sample Plot (PSP). The models were calibrated using the NFI-data and tested against the PSP-data. Calibration was done both per country and for all countries simultaneously, thus yielding country-specific and generic parameter distributions. We assessed model performance by sampling from prior and posterior distributions and comparing the growth predictions of these samples to the observations at the PSPs. We found that BC reduced uncertainties strongly in all but the most complex model. Surprisingly, country-specific BC did not lead to clearly better within-country predictions than generic BC. BMC identified the BRIDGING model, which is of intermediate complexity, as the most plausible model before calibration, with 4C taking its place after calibration. In this BMC, model plausibility was quantified as the relative probability of a model being correct given the information in the PSP-data. We discuss how the method of model initialisation affects model performance. Finally, we show how BMA affords a robust way of predicting forest growth that accounts for both parametric and model structural uncertainty. <s> BIB007 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Quantifying uncertainty of model-averaged predictions <s> We develop an approach to evaluating frequentist model averaging procedures by considering them in a simple situation in which there are two-nested linear regression models over which we average. We introduce a general class of model averaged confidence intervals, obtain exact expressions for the coverage and the scaled expected length of the intervals, and use these to compute these quantities for the model averaged profile likelihood (MPI) and model-averaged tail area confidence intervals proposed by D. Fletcher and D. Turek. We show that the MPI confidence intervals can perform more poorly than the standard confidence interval used after model selection but ignoring the model selection process. The model-averaged tail area confidence intervals perform better than the MPI and postmodel-selection confidence intervals but, for the examples that we consider, offer little over simply using the standard confidence interval for θ under the full model, with the same nominal coverage. <s> BIB008
So far, we have shown that model averaging can produce predictions with a smaller error than any of the contributing models by averaging away their variance and bias. Those gains, however, generally decrease with increasing covariance of the individual model predictions, and increasing mean bias of the contributing models. Moreover, weighted averaging allows reducing the weight of models poorly supported by data, but at the expense of introducing additional variance in the average, induced by the weight estimation. Besides having an estimate with low error, the second goal of most statistical methods is to provide a measure of (un)certainty of that estimate. The nature of this measure differs between tactical, Bayesian, and frequentist approaches. Tactical aproaches, such as machine learning, are usually satisfied with providing an estimate of predictive error on new data, typically obtained through cross-validation. This procedure can be directly extended to model-averaged predictions. For Bayesian and frequentist methods, the issue of extending the conventional methods for estimating uncertainty to model-averaging is somewhat more complicated. Bayesian methods quantify uncertainty via the posterior distribution, which can be summarized by a Bayesian credible interval. One would interpret a 95% credible interval as displaying a 95% certainty for the true value to be contained in the interval. Frequentist methods traditionally provide a confidence interval. Under repeated sampling of new data sets under identical conditions, a correctly defined 95% confidence interval should contain the true value in 95% of the cases. To construct a frequentist confidence interval for a modelaveraged prediction, we have to ask ourselves how this modelaveraged prediction will spread around the true value under repeated sampling. Fortunately, we have already derived this result in Eqs. 1-5. For simple cases, we can directly convert this into a confidence interval. For example, for an unbiased average, with uncorrelated models of equal weight and variance, the standard deviation of the average, and thus its confidence interval, should decrease with one over the square root of the number of contributing models, times the confidence interval of the single models. In general, however, the calculation of the confidence interval of the average will have to take the confidence intervals of all contributing models, as well as their weights, covariance and bias into account. BIB001 proposed a simplification of Eq. 5, which considers bias and variance of the averaged models (for derivation see Burnham and Anderson 2002:159-162  Misspecification bias of model m is computed as thus assuming (explicitly on page 604 of of BIB001 ) that the averaged point estimate e Y is unbiased and can hence be used to compute the bias of the individual predictions. This assumption can be visualized in Fig. 2 as the situation where the empty triangles always sit right on top of "truth." This assumption is problematic, as it cannot be met by unidirectionally biased model predictions, nor when weights w m fail to get the weighting exactly right and thus e Y remains biased. Less problematically, BIB001 also assumed that predictions from different models are perfectly correlated, making the covariance term as large as possible, and variance estimation conservative. The distribution theory behind this approach has been criticized as "not (even approximately) correct" (Claeskens and Hjort 2008:207) , but shown to work well in simulations (Lukacs et al. 2010, Fletcher and BIB003 . Improving on Eq. 6 requires knowledge of the correlation of model predictions q mm 0 (Eq. 5). The key problem is that there 490 DORMANN Vol. 88, No. 4 is no analytical way to compute q mm 0 . Bootstrapping, although computationally costly, offers a good solution to this problem. While the obstacles to calculate confidence intervals for model-averaged predictions may seem somewhat discouraging, it should be noted that alternatives to model averaging do not necessarily fare better. Predictions from a selected single-best model always underestimate the true prediction error (e.g., BIB002 . The reason is that the uncertainty about which model is correct is not included in this final prediction: we predict as if we had not carried out model selection but had known from the beginning which model would be the best (as if the model had been "prescribed": Harrell 2001). Thus, even if we were able to choose, from our model set M, the model closest to truth, we would still need to adjust the confidence distribution for model selection; and a perfect adjustment was analytically shown not to exist BIB008 . Accordingly, simulation studies have suggested that model averaging may improve coverage  BIB002 BIB006 , presumably because the process of averaging allows us to take into account model uncertainty BIB004 ). Yet, given the diversity of approaches to computing model weights encountered in Approaches to Estimating Model-Averaging Weight, these studies cannot be seen as conclusive, only as suggestive, for the improvement of nominal coverage using model averaging. For example BIB005 and BIB007 explore how model averaging can improve the tail areas of the confidence distribution. These two studies, however, as well as those cited before, assumed that the full model, referring to the model that includes all sub-models prior to any model selection (see Appendix S1.3), is not in the set. The approach by BIB005 was re-analyzed by BIB008 . The key finding of this latter study is that the full model coverage was still superior to all other model averaging approaches, suggesting that the full model should currently be kept in mind, both for inference, minimal bias and correct prediction intervals (see also Harrell 2001:59) . Such findings sit uncomfortably with the bias-variance trade-off  ), which states that overly complex models have poor predictive performance; and indeed the full model has high prediction variance. Regrettably, such reasoning cannot be extended in an obvious way to non-nested models, process models, or machine learning models. Here, model averaging seems without alternative for propagating model selection uncertainty into prediction uncertainty more fairly. Our final option to quantify uncertainty, the Bayesian credible interval, can be interpreted as a mixture distribution. In a two-step process, the model weights first determine the probability of any model to be correct, and the uncertainty of each model is then mixed additively into a averaged uncertainty. If the predictions of all individual models are identical, the final distribution will remain the same; from the perspective of Eq. 5, this is identical to assuming that the average models are perfectly correlated, although the logical motivation for the mixing is different. If predictions differ widely, e.g., due to bias, the mixed confidence distribution will be much wider and possibly multi-modal. To illustrate the various Bayesian and frequentist options, we calculated predictive uncertainties and coverage for four different options for a set of simple linear regressions in Fig. 4: 1) Make the assumption that model-averaged predictions are unbiased. Use bootstrapping to estimate covariances of predictions for each model. From these estimates, compute prediction variance according to Eq. 5. This solution is computer-intensive, but it takes into account covariance of model predictions. On the other hand, it cannot account for bias, and should thus not be used when bias of the estimator is suspected, for example from cross-validation. 2) Make the assumption that model-averaged predictions are unbiased. Use BIB001 's approach (Eq. 6). This will yield wider estimates than option 1, because assumptions about bias and correlation are more conservative. 3) Use a mixture distribution to compute the confidence distribution of the average, assuming effectively that predictions from different models are perfectly correlated, but possibly biased. 4) Fit the full model (if available) and use its confidence distribution, which can rarely be improved on BIB008 . When averaging models with largely independent (i.e., uncorrelated) predictions, only the bootstrap-estimated covariance matrix (option 1 above) will also compute lower variances (according to Eq. 4). In our example ( Fig. 4 , see Data S1 for details), "propagation" produced the tightest confidence interval (and hence lowest coverage), followed by "Buckland" and "mixing". However, neither of these confidence intervals seemed large enough, as all had too low coverage (suggesting model bias to be relevant in this example). Only the full model produces accurate confidence intervals and coverage. Further simulations along these lines will have to show how these approaches perform for more complex models and situations.