Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Bayesian model weights <s> Abstract In the theory of linear models, the concept of degrees of freedom plays an important role. This concept is often used for measurement of model complexity, for obtaining an unbiased estimate of the error variance, and for comparison of different models. I have developed a concept of generalized degrees of freedom (GDF) that is applicable to complex modeling procedures. The definition is based on the sum of the sensitivity of each fitted value to perturbation in the corresponding observed value. The concept is nonasymptotic in nature and does not require analytic knowledge of the modeling procedures. The concept of GDF offers a unified framework under which complex and highly irregular modeling procedures can be analyzed in the same way as classical linear models. By using this framework, many difficult problems can be solved easily. For example, one can now measure the number of observations used in a variable selection process. Different modeling procedures, such as a tree-based regression and a ... <s> BIB001 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Bayesian model weights <s> This paper reviews the Bayesian approach to model selection and model averaging. In this review, I emphasize objective Bayesian methods based on noninformative priors. I will also discuss implementation details, approximations, and relationships to other methods. Copyright 2000 Academic Press. <s> BIB002 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Bayesian model weights <s> We discuss Bayesian methods for model averaging and model selection among Bayesian-network models with hidden variables. In particular, we examine large-sample approximations for the marginal likelihood of naive-Bayes models in which the root node is hidden. Such models are useful for clustering or unsupervised learning. We consider a Laplace approximation and the less accurate but more computationally efficient approximation known as the Bayesian Information Criterion (BIC), which is equivalent to Rissanen's (1987) Minimum Description Length (MDL). Also, we consider approximations that ignore some off-diagonal elements of the observed information matrix and an approximation proposed by Cheeseman and Stutz (1995). We evaluate the accuracy of these approximations using a Monte-Carlo gold standard. In experiments with artificial and real examples, we find that (1) none of the approximations are accurate when used for model averaging, (2) all of the approximations, with the exception of BIC/MDL, are accurate for model selection, (3) among the accurate approximations, the Cheeseman–Stutz and Diagonal approximations are the most computationally efficient, (4) all of the approximations, with the exception of BIC/MDL, can be sensitive to the prior distribution over model parameters, and (5) the Cheeseman–Stutz approximation can be more accurate than the other approximations, including the Laplace approximation, in situations where the parameters in the maximum a posteriori configuration are near a boundary. <s> BIB003 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Bayesian model weights <s> Bayesian estimation methods are used to fit an age- and sex-structured population model to available data on abundance and stag eproportions (i.e. calves/mature animals in the population) for the Bering-Chukchi-Beaufort Seas stock of bowhead whales ( Balaena mysticetus). The analyses consider three alternative population modelling approaches: (1) modelling the entire population trajectory from 1848, using the ‘backwards’ method where the trajectory is back-calculated based on assigning a prior distribution to recent abundance ; (2) modelling only the recent population trajectory, using the ‘forwards from recent abundance’ method, where the population is projected forwards from a recent year and the abundance in that year is not assumed to be at carrying capacity; and (3) a version of (2) that ignores density-dependence. The ‘backwards’ method leads to more precise estimates of depletion level. In contrast, the ‘forwards from r ecent abundance’ method provides an alternative way of calculating catch-related quantities without having to assume that the catch record is known exactly from 1848 to the present, or having to assume that carrying capacity has not changed since 1848. Not only are all three models able to fit the abundance data well, but each is also able to remain consistent with available estimates of adult surviv al and age of sexual maturity. Sensitivity to the stage-proportion data and the prior distributions for the life history parameters indicates that use of the 1985 stage-proportion data has the greatest effect on the results, and that those data are less consistent with data on trends in abundance and age of sexual maturity. The analyses indicate that the population has approximately doubled in size since 1978, and the ‘backwards’ analyses suggest that the population may be approaching carrying capacity, although there is no obvious sign in the data that the popula tion growth rate has slowed. Bayes factors are calculated to compare model fits to the data. However, there is no evidence for selecting on e model over another, and furthermore, the models considered in this study result in different posterior distributions for quantities of int erest to management. Posterior model probabilities are therefore calculated and used as weights to construct Bayesian model-averaged pos terior distributions for outputs shared among models to take this ambiguity into account. This study represents the first attempt to e xplicitly quantify model uncertainty when conducting a stock assessment of bowhead whales. <s> BIB004 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Bayesian model weights <s> Statistical thinking in wildlife biology and ecology has been profoundly influenced by the introduction of AIC (Akaike's information criterion) as a tool for model selection and as a basis for model averaging. In this paper, we advocate the Bayesian paradigm as a broader framework for multimodel inference, one in which model averaging and model selection are naturally linked, and in which the performance of AIC-based tools is naturally evaluated. Prior model weights implicitly associated with the use of AIC are seen to highly favor complex models: in some cases, all but the most highly parameterized models in the model set are virtually ignored a priori. We suggest the usefulness of the weighted BIC (Bayesian information criterion) as a computationally simple alternative to AIC, based on explicit selection of prior model probabilities rather than acceptance of default priors associated with AIC. We note, however, that both procedures are only approximate to the use of exact Bayes factors. We discuss and illustrate technical difficulties associated with Bayes factors, and suggest approaches to avoiding these difficulties in the context of model selection for a logistic regression. Our example highlights the predisposition of AIC weighting to favor complex models and suggests a need for caution in using the BIC for computing approximate posterior model weights. <s> BIB005 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Bayesian model weights <s> Approximate Bayesian computation (ABC) methods can be used to evaluate posterior distributions without having to calculate likelihoods. In this paper, we discuss and apply an ABC method based on sequential Monte Carlo (SMC) to estimate parameters of dynamical models. We show that ABC SMC provides information about the inferability of parameters and model sensitivity to changes in parameters, and tends to perform better than other ABC approaches. The algorithm is applied to several well-known biological systems, for which parameters and their credible intervals are inferred. Moreover, we develop ABC SMC as a tool for model selection; given a range of different mathematical descriptions, ABC SMC is able to choose the best model using the standard Bayesian model selection apparatus. <s> BIB006 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Bayesian model weights <s> Computation of the marginal likelihood from a simulated posterior distribution is central to Bayesian model selection but is computationally difficult. I argue that the marginal likelihood can be reliably computed from a posterior sample by careful attention to the numerics of the probability integral. Posing the expression for the marginal likelihood as a Lebesgue integral, we may convert the harmonic mean approximation from a sample statistic to a quadrature rule. As a quadrature, the harmonic mean approximation suffers from enormous truncation error as consequence . In addition, I demonstrate that the integral expression for the harmonic-mean approximation converges slowly at best for high-dimensional problems with uninformative prior distributions. These observations lead to two computationally-modest families of quadrature algorithms that use the full generality sample posterior but without the instability. The first algorithm automatically eliminates the part of the sample that contributes large truncation error. The second algorithm uses the posterior sample to assign probability to a partition of the sample space and performs the marginal likelihood integral directly. This eliminates convergence issues. The first algorithm is analogous to standard quadrature but can only be applied for convergent problems. The second is a hybrid of cubature: it uses the posterior to discover and tessellate the subset of that sample space was explored and uses quantiles to compute a representive field value. Neither algorithm makes strong assumptions about the shape of the posterior distribution and neither is sensitive outliers. [abridged] <s> BIB007 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Bayesian model weights <s> Ecology Letters (2011) 14: 816–827 ::: ::: ::: ::: Abstract ::: ::: Statistical models are the traditional choice to test scientific theories when observations, processes or boundary conditions are subject to stochasticity. Many important systems in ecology and biology, however, are difficult to capture with statistical models. Stochastic simulation models offer an alternative, but they were hitherto associated with a major disadvantage: their likelihood functions can usually not be calculated explicitly, and thus it is difficult to couple them to well-established statistical theory such as maximum likelihood and Bayesian statistics. A number of new methods, among them Approximate Bayesian Computing and Pattern-Oriented Modelling, bypass this limitation. These methods share three main principles: aggregation of simulated and observed data via summary statistics, likelihood approximation based on the summary statistics, and efficient sampling. We discuss principles as well as advantages and caveats of these methods, and demonstrate their potential for integrating stochastic simulation models into a unified framework for statistical modelling. <s> BIB008 </s> Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference <s> Bayesian model weights <s> ABSTRACTGeneralized degrees of freedom (GDF), as defined by Ye (1998 JASA 93:120–131), represent the sensitivity of model fits to perturbations of the data. Such GDF can be computed for any statist... <s> BIB009
Theory.-Bayes' formula can be applied to choosing among models in much the same way as to parameter values BIB002 . To perform inference with multiple models and their parameters at the same time, one can write down the joint posterior probability PðM i ; H i jDÞ of model M i with parameter vector H i , given the observed data D, as where LðDjM i ; H i Þ is the likelihood of model M i , pðH i Þ is the prior distribution of the parameters of the respective model M i , and pðM i Þ is the prior weight on model M i . In practice, one is often interested in some simplified statistics from this distribution, such as the model with the highest posterior model probability, or the distribution of a prediction including model selection uncertainty. To obtain this information, we can marginalize (i.e., integrate) over parameter space, or marginalize over model space, respectively. If we marginalize over parameter space, we obtain posterior model weights that represent the relative probability of each model (whilst marginalizing over model space yields averaged parameters, which we shall not address here). We can calculate these weights as the marginal likelihood of each model, defined as the average of Eq. 7 across all k parameters for any given model From the marginal likelihood, we can compare models via the Bayes factor, defined as the ratio of their marginal likelihoods (e.g.,   with the multiple integral now pulled together for notational convenience. For more than two models, however, it is more useful to standardize this quantity across all models in question, calculating a Bayesian posterior model weight pðM i jDÞ (including model priors pðM i Þ: Kass and Raftery 1995) as Vol. 88, No. 4 Estimation in practice.-While the definition of Bayesian model weights and averaged parameters is straightforward, the estimation of these quantities can be challenging. In practice, there are two options to numerically estimate the quantities defined above, both with caveats. The first option is to sample directly from the joint posterior (Eq. 7) of the models and the parameters. Basic algorithms such as rejection sampling can do that without any modification (e.g., BIB006 ), but they are inefficient for higher-dimensional parameter spaces. More sophisticated algorithms such as MCMC and SMC (see BIB008 , for a basic review) require modifications to deal with the issue of different number of parameters when changing between models. Such modifications (mostly the reversible-jump MCMCs, rjMCMC; Green 1995; see Appendix S1.5.1) are often difficult to program, tune and generalize, which is the reason why they are typically only applied in specialized, well-defined settings. The posterior model probabilities of the rjMCMC are estimated as the proportion of time the algorithm spent with each model, measured as the number of iterations the algorithm drew a particular model divided by the total number of iterations. The second option is to approximate the marginal likelihood in Eq. 8 of each model independently, renormalize that into weights, and then average predictions based on these weights. The challenge here is to get a stable approximation of the marginal likelihood, which can be problematic BIB007 ; see Appendix S1.5.1). Still, because of the relatively simple implementation, this approach is a more common choice than rjMCMC (e.g., BIB004 . Influence of priors.-A problem for the computation of model weights when performing Bayesian inference across multiple models is the influence of the choice of parameter priors, especially "uninformative" ones (see BIB003 Heckerman 1997, Hoeting et al. 1999: section 5) . The challenge arises because in Eqs. 8 and 9 the prior density pðh i Þ enters the marginal likelihood, and hence the Bayes factor, multiplicatively. This has the somewhat unintuitive consequence that increasing the width of an uninformative parameter prior will linearly decrease the model's marginal likelihood (e.g., BIB005 . That Bayesian model weights are strongly dependent on the width of the prior choice has sparked discussion of the appropriateness of this approach in situations with uninformative priors. For example, in situations where multiple nested models are compared, the width of the uninformative prior may completely determine the complexity of models that are being selected. One suggestion that has been made is to not at all perform multi-model inference with uninformative priors, but at least additional corrections are necessary to apply Bayes factors weights (O'Hagan 1995, Berger and  . One such correction is to calibrate the model on a part of the data first, use the result as new priors and then perform the analysis described above (intrinsic Bayes factor: Berger and Pericchi 1996, fractional Bayes factor: O'Hagan 1995). If enough data are available so that the likelihood is Notes: AIC, Akaike information criterion; WAIC, widely applicable information criterion; BIC, Bayesian information criterion. †Does this method require a maximum-likelihood fit and/or number of parameters (p m ) of the model? Typically these two are linked, since maximum-likelihood approaches typically employ the general linear model (GLM), which provides both information. ‡See also Appendix for details and case studies in Data SI for examples of implementation in R. §While nonparametric models have no readily extractable number of parameters, a Generalized Degrees of Freedom-approach could be used to compute them BIB001 . Similarly, but more efficiently, cross-validation can be used to estimate the effective number of parameters BIB009 . ¶Implemented in MuMIn as part of this publication. #http://users.stat.umn.edu/~sandy/courses/8053/handouts/Aaron/ARMS/.